Oct 31 22:48:49.659930 ip-10-0-133-222 systemd[1]: Starting Kubernetes Kubelet... Oct 31 22:48:50.103203 ip-10-0-133-222 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:48:50.103203 ip-10-0-133-222 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 22:48:50.103203 ip-10-0-133-222 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:48:50.103203 ip-10-0-133-222 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:48:50.111429 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111342 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.111429 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111423 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.111429 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111427 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.111429 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111430 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.111429 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111433 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.111429 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111437 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111440 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111443 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111446 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111449 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111451 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111454 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111456 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111459 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111461 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111464 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111467 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111470 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111472 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111475 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111478 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111481 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111484 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111487 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.111708 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111489 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111492 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111494 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111506 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111509 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111512 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111514 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111517 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111521 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111524 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111526 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111529 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111532 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111534 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111537 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111540 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111544 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111548 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.112143 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111552 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111555 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111558 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111561 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111563 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111565 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111570 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111574 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111577 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111580 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111588 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111591 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111594 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111597 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111599 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111602 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111605 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111607 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111610 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.112563 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111612 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111615 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111618 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111620 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111623 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111627 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111630 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111633 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111635 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111639 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111641 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111644 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111647 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111649 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111652 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111655 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111672 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111677 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111680 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111683 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.113030 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111685 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111688 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111692 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111695 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111698 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111700 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111703 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111706 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111708 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111711 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111714 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111716 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111719 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111722 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111724 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111727 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111729 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111732 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111735 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111738 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.113497 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111741 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111744 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111746 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111750 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.111753 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112510 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112516 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112520 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112522 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112525 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112528 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112530 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112533 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112536 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112538 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112541 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112544 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112547 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112549 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112552 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.113981 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112555 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112557 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112560 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112563 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112565 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112568 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112571 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112574 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112577 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112579 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112583 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112585 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112588 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112590 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112593 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112596 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112598 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112601 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112604 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112606 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.114486 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112609 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112612 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112615 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112617 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112620 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112623 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112625 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112628 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112631 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112634 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112636 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112639 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112641 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112644 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112646 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112649 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112652 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112655 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112675 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112679 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.114975 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112681 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112684 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112687 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112690 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112692 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112695 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112698 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112700 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112703 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112705 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112708 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112710 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112713 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112715 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112718 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112728 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112731 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112734 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112736 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.115449 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112739 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112741 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112744 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112747 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112752 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112756 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112758 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112761 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112771 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112774 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112776 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112779 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112781 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112784 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112787 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112790 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112793 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112795 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112798 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112801 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.115910 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112803 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112806 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112808 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112811 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112814 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112816 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112819 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112823 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112826 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112829 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112832 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.112834 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113313 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113318 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113324 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113328 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113331 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113333 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113336 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.116381 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113339 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113342 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113345 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113348 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113351 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113353 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113356 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113359 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113362 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113364 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113367 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113370 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113372 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113375 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113377 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113380 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113382 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113385 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113388 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.116853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113390 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113393 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113396 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113398 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113401 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113404 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113407 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113409 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113412 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113415 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113417 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113420 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113422 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113425 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113428 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113431 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113433 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113436 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113438 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113441 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.117294 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113443 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113445 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113448 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113451 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113453 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113455 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113458 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113461 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113463 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113465 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113468 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113470 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113475 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113478 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113481 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113484 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113487 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113489 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113492 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113494 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.117790 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113497 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113500 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113503 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113506 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113509 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113511 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113514 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113517 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113519 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113522 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113524 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113527 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113529 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113532 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113534 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113537 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113539 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113542 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113544 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113546 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.118254 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113549 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113551 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113554 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113556 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113559 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113561 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113564 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113567 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113572 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113577 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113580 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113584 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113586 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113589 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113591 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113600 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113603 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113605 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113608 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.118738 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.113610 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114725 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114736 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114741 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114746 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114750 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114754 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114758 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114763 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114766 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114769 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114772 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114776 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114779 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114782 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114785 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114788 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114791 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114794 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114803 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114806 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114809 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114812 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114816 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 22:48:50.119209 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114820 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114823 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114826 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114830 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114833 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114842 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114845 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114848 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114851 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114856 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114859 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114862 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114865 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114868 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114871 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114879 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114882 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114885 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114888 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114891 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114896 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114899 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114904 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114907 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114911 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 22:48:50.119846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114914 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114917 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114920 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114923 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114925 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114928 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114933 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114936 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114939 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114942 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114945 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114948 2576 flags.go:64] FLAG: --help="false" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114951 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114955 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114959 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114961 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114965 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114968 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114971 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114974 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114977 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114980 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114983 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 22:48:50.120445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114986 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114989 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114992 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114995 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.114999 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115002 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115006 2576 flags.go:64] FLAG: --lock-file="" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115009 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115012 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115015 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115021 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115024 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115027 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115030 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115033 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115036 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115039 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115042 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115046 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115050 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115053 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115056 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115059 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115063 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115066 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 22:48:50.121018 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115069 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115072 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115075 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115081 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115085 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115088 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115091 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115094 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115096 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115100 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115103 2576 flags.go:64] FLAG: --port="10250" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115107 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115110 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d0921356442108bb" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115113 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115116 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115120 2576 flags.go:64] FLAG: --register-node="true" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115123 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115126 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115129 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115132 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115135 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115139 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115142 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115145 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115148 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115151 2576 flags.go:64] FLAG: --runonce="false" Oct 31 22:48:50.121794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115154 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115157 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115160 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115163 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115165 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115169 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115172 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115175 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115178 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115181 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115184 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115187 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115190 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115193 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115195 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115201 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115204 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115208 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115212 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115215 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115218 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115221 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115224 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115227 2576 flags.go:64] FLAG: --v="2" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115231 2576 flags.go:64] FLAG: --version="false" Oct 31 22:48:50.122889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115235 2576 flags.go:64] FLAG: --vmodule="" Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115239 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115242 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115374 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115378 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115381 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115384 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115387 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115389 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115392 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115395 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115398 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115400 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115404 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115407 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115410 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115413 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115415 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115418 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.123990 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115421 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115423 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115426 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115428 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115431 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115435 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115438 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115441 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115443 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115446 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115448 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115451 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115454 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115457 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115460 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115462 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115465 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115467 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115470 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.124447 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115473 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115477 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115481 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115485 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115488 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115491 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115493 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115496 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115499 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115503 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115506 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115508 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115511 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115513 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115516 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115519 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115521 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115524 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115529 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115532 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.125059 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115534 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115537 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115540 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115542 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115545 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115547 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115550 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115553 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115555 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115558 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115560 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115563 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115565 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115568 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115571 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115574 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115576 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115579 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115582 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115584 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.125972 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115587 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115591 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115594 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115596 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115599 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115601 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115604 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115607 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115609 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115612 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115615 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115618 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115621 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115623 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115626 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115629 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115631 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115634 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115636 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115639 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.126843 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115642 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115644 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115647 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115650 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115652 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115655 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115671 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115675 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115678 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115680 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115683 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.115711 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115757 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115763 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115767 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115769 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.127325 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115772 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115775 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115778 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115780 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115783 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115785 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115788 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115793 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115795 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115798 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115801 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115803 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115806 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115808 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115811 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115814 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115817 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115819 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115822 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115824 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.127858 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115827 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115830 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115832 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115835 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115837 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115840 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115845 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115848 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115852 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115854 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115857 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115860 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115863 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115865 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115868 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115871 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115874 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115876 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115880 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.128682 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115883 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115886 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115889 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115891 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115894 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115896 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115899 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115903 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115905 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115908 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115910 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115913 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115916 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115919 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115923 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115926 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115930 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115933 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115936 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.129313 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115938 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115941 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115945 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115948 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115951 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115954 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115956 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115959 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115961 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115964 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115966 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115969 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115972 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115975 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115977 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115980 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115982 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115985 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115987 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.129938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115990 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115992 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.115996 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116001 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116004 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116006 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116009 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116012 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116014 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116017 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116019 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116022 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116024 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116026 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116029 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116034 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116037 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116039 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116042 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116045 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.130421 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116047 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116050 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116052 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.116055 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.123921 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.123940 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.124778 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.124791 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124877 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124885 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124890 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124895 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124900 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124904 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124908 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124913 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124917 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124924 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124930 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.130934 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124936 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124941 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124946 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124950 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124955 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124959 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124964 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124968 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124972 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124978 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124982 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124988 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124993 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.124998 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125002 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125006 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125010 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125015 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125019 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.131409 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125024 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125029 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125034 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125038 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125043 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125047 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125051 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125055 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125059 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125064 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125068 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125072 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125076 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125082 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125088 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125093 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125097 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125102 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125106 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.131938 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125111 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125115 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125119 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125124 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125128 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125132 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125137 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125143 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125148 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125152 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125157 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125161 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125166 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125170 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125174 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125178 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125183 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125188 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125192 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.132386 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125197 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125201 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125205 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125209 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125213 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125217 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125222 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125226 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125231 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125235 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125244 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125248 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125252 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125257 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125261 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125266 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125270 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125275 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125280 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.132901 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125284 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125289 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125293 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125297 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125302 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125306 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125311 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125315 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125319 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125323 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125328 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125332 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125336 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125341 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125345 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125349 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125354 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.133872 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125358 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.125398 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125610 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125617 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125622 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125627 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125633 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125637 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125643 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125648 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125652 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125656 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125678 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125682 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125686 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125690 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.134501 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125694 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125700 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125704 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125709 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125713 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125718 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125721 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125726 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125730 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125734 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125738 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125742 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125747 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125751 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125755 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125760 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125767 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125773 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125777 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.134913 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125782 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125786 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125790 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125794 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125799 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125803 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125807 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125811 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125816 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125820 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125824 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125828 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125833 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125837 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125841 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125847 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125852 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125856 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125861 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125865 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.135387 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125870 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125874 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125878 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125882 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125886 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125890 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125895 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125899 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125903 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125907 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125911 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125915 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125920 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125924 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125928 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125932 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125939 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125946 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125951 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.135898 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125956 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125960 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125965 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125969 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125973 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125977 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125980 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125985 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125989 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125994 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.125998 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126002 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126007 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126012 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126016 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126020 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126025 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126029 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126033 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126037 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.136339 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126042 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126046 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126051 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126055 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126059 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126063 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126068 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126072 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126076 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126081 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126085 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126089 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126094 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126099 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.126137 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:48:50.136853 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126188 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126195 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126200 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126205 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126209 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126214 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126218 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126223 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126228 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126232 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126236 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126240 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126244 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126248 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126252 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126257 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126261 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126265 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126270 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126274 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:48:50.137205 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126278 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126283 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126287 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126291 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126295 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126300 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126304 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126308 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126313 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126317 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126322 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126326 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126330 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126334 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126341 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126347 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126352 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126357 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126361 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:48:50.137699 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126365 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126371 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126375 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126379 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126384 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126388 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126392 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126397 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126401 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126405 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126409 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126415 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126419 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126423 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126428 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126432 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126437 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126441 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126445 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126449 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:48:50.138140 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126454 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126458 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126462 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126466 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126470 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126474 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126479 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126483 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126487 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126491 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126495 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126499 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126503 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126508 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126514 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126520 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126527 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126532 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126536 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126540 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:48:50.138948 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126547 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126551 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126556 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126560 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126564 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126568 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126572 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126577 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126581 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126585 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126589 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126594 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126598 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126603 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126607 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126611 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126615 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126620 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126624 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126628 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:48:50.139642 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126632 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126636 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126640 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126645 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126649 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126653 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:50.126671 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.126746 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.129831 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.130942 2576 server.go:1014] "Starting client certificate rotation" Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.131037 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.131077 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 22:48:50.140199 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.133435 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:48:50.140507 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.140486 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:48:50.156277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.156260 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 22:48:50.161318 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.161295 2576 log.go:25] "Validated CRI v1 image API" Oct 31 22:48:50.162149 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.162128 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 22:48:50.166825 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.166806 2576 fs.go:135] Filesystem UUIDs: map[5e2f145a-ebf5-4636-a5f4-1f99e29d9d6b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8102c9d4-2163-4ec4-96ca-e2899c91ab97:/dev/nvme0n1p4] Oct 31 22:48:50.166891 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.166825 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 22:48:50.171148 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.171006 2576 manager.go:217] Machine: {Timestamp:2026-10-31 22:48:50.170520422 +0000 UTC m=+0.383758824 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098973 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec285524684e78a4163432431ad8a1a8 SystemUUID:ec285524-684e-78a4-1634-32431ad8a1a8 BootID:4adba19c-fefa-4c74-8808-bde9684fe012 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:f1:92:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:f1:92:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:24:a1:3a:b6:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 22:48:50.171148 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.171138 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 22:48:50.171258 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.171249 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 22:48:50.174357 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.174333 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 22:48:50.174481 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.174360 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-222.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 22:48:50.174522 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.174483 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 22:48:50.174522 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.174491 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 22:48:50.174522 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.174507 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:48:50.174522 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.174519 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:48:50.175759 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.175749 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 22:48:50.176014 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.175997 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 22:48:50.180268 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180257 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 22:48:50.180374 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180365 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 22:48:50.180404 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180395 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 22:48:50.180437 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180406 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 22:48:50.180437 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180416 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 22:48:50.180437 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180427 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 22:48:50.180437 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180435 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 22:48:50.180767 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.180756 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:48:50.183268 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.183248 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 22:48:50.190341 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.190328 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:48:50.191247 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191235 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 22:48:50.191294 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191265 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 22:48:50.191417 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191407 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 22:48:50.191445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191423 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 22:48:50.191445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191429 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 22:48:50.191445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191435 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 22:48:50.191445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191440 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 22:48:50.191445 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.191446 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 22:48:50.192288 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.192279 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 22:48:50.192324 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.192289 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 22:48:50.192324 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.192296 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 22:48:50.192324 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.192303 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 22:48:50.192324 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.192311 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 22:48:50.192324 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.192320 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 22:48:50.193862 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.193850 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 22:48:50.193902 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.193875 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 22:48:50.198550 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.198532 2576 server.go:1263] "Started kubelet" Oct 31 22:48:50.198683 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.198614 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 22:48:50.198760 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.198689 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 22:48:50.198857 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.198778 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 22:48:50.199425 ip-10-0-133-222 systemd[1]: Started Kubernetes Kubelet. Oct 31 22:48:50.199606 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.199584 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-222.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 22:48:50.199606 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.199594 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-222.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 22:48:50.199857 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.199836 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 22:48:50.200173 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.200157 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 22:48:50.200764 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.200750 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 22:48:50.207599 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.207578 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 22:48:50.208148 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.208117 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 22:48:50.209432 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.209417 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 22:48:50.209523 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.209506 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.210202 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210185 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 22:48:50.210202 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210184 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 22:48:50.210316 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210213 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 22:48:50.210316 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210309 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 22:48:50.210384 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210319 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 22:48:50.210426 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210414 2576 factory.go:153] Registering CRI-O factory Oct 31 22:48:50.210454 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210429 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 22:48:50.210503 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210493 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 22:48:50.210503 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210503 2576 factory.go:55] Registering systemd factory Oct 31 22:48:50.210605 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210509 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 22:48:50.210605 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210521 2576 factory.go:103] Registering Raw factory Oct 31 22:48:50.210605 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.210531 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 22:48:50.211030 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.209804 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.us-east-2.compute.internal.18e3bedc57c9387c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 22:48:50.198476924 +0000 UTC m=+0.411715327,LastTimestamp:2026-10-31 22:48:50.198476924 +0000 UTC m=+0.411715327,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.us-east-2.compute.internal,}" Oct 31 22:48:50.211576 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.211559 2576 manager.go:319] Starting recovery of all containers Oct 31 22:48:50.223516 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.223490 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-222.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 22:48:50.223775 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.223756 2576 manager.go:324] Recovery completed Oct 31 22:48:50.227597 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.226528 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54ws9" Oct 31 22:48:50.227597 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.227507 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 22:48:50.231767 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.231745 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 22:48:50.231767 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.231752 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.233453 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.233433 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.233512 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.233472 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.233512 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.233485 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.233995 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.233978 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 22:48:50.233995 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.233991 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 22:48:50.233995 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.233994 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.234151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.234009 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 22:48:50.235124 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.235046 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.us-east-2.compute.internal.18e3bedc59deeeca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-222.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 22:48:50.233454282 +0000 UTC m=+0.446692686,LastTimestamp:2026-10-31 22:48:50.233454282 +0000 UTC m=+0.446692686,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.us-east-2.compute.internal,}" Oct 31 22:48:50.235463 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.235446 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.235534 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.235471 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.235534 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.235484 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.236214 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.236200 2576 policy_none.go:50] "Start" Oct 31 22:48:50.236214 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.236216 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 22:48:50.236298 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.236226 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 22:48:50.238256 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.238245 2576 policy_none.go:44] "Start" Oct 31 22:48:50.238481 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.238469 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-54ws9" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.275234 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.277791 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.278480 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.278499 2576 server.go:87] "Starting device plugin registration server" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.278767 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.278779 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.278869 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.278943 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.278952 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.279552 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.279596 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.282888 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.284028 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.284044 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.284058 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.284085 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 22:48:50.288727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.288726 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:48:50.379511 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.379470 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.380490 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.380465 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.380553 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.380498 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.380553 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.380509 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.380553 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.380536 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.384936 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.384918 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal"] Oct 31 22:48:50.385010 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.384978 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.386389 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.386376 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.386471 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.386404 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.386471 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.386417 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.388652 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.388637 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.388807 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.388793 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.388855 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.388822 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.389425 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.389408 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.389466 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.389436 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.389466 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.389447 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.389555 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.389408 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.389555 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.389516 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.389555 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.389534 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.391536 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.391519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.391628 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.391543 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:48:50.391746 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.391732 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.391814 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.391752 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-222.us-east-2.compute.internal\": node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.392275 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.392255 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:48:50.392356 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.392279 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:48:50.392356 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.392293 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:48:50.411724 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.411706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.411818 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.411729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.411818 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.411746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70adfcc128189ec0c0f585313e4a644-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"a70adfcc128189ec0c0f585313e4a644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.417190 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.417168 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.419749 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.419732 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.474249 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.474230 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.512687 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.512653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.512766 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.512691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.512766 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.512711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70adfcc128189ec0c0f585313e4a644-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"a70adfcc128189ec0c0f585313e4a644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.512832 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.512766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.512832 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.512769 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.512832 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.512819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70adfcc128189ec0c0f585313e4a644-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"a70adfcc128189ec0c0f585313e4a644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.574790 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.574772 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.675276 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.675220 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.722810 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.722792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.724946 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:50.724923 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:50.775618 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.775498 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:50.876009 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:50.875976 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:52.606413 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:52.606388 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:52.706985 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:52.706962 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:52.729057 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.729032 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:48:52.760440 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.760421 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 22:48:52.760808 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.760531 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 22:48:52.760808 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.760584 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 22:48:52.807745 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:52.807710 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:52.856829 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.856776 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:48:52.874248 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.874214 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 22:43:51 +0000 UTC" deadline="2028-07-16 02:32:41.923924236 +0000 UTC" Oct 31 22:48:52.874248 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:52.874246 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14955h43m49.049681028s" Oct 31 22:48:52.908379 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:52.908348 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:53.008772 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.008741 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:53.109318 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.109254 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:48:53.200576 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.200536 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:48:53.218635 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:53.218598 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87261ed136f2c9536dc41bf58fda7ff.slice/crio-efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9 WatchSource:0}: Error finding container efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9: Status 404 returned error can't find the container with id efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9 Oct 31 22:48:53.219354 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:53.219332 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70adfcc128189ec0c0f585313e4a644.slice/crio-25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894 WatchSource:0}: Error finding container 25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894: Status 404 returned error can't find the container with id 25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894 Oct 31 22:48:53.227442 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.227428 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:48:53.240170 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.240154 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:53.247668 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.247652 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:48:53.248697 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.248685 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:48:53.260237 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.260216 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:48:53.811423 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.811396 2576 apiserver.go:52] "Watching apiserver" Oct 31 22:48:53.821113 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.819885 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 22:48:53.821113 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.820601 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-56mmn","kube-system/konnectivity-agent-jpmrk","openshift-dns/node-resolver-sk99x","openshift-multus/multus-8d5v6","openshift-ovn-kubernetes/ovnkube-node-f6f9f","kube-system/global-pull-secret-syncer-xrfs4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj","openshift-image-registry/node-ca-58lh2","openshift-multus/multus-additional-cni-plugins-gbvzb","openshift-multus/network-metrics-daemon-cz8ch","openshift-network-operator/iptables-alerter-q85ln","kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-gljgz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal"] Oct 31 22:48:53.823675 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.823282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.827003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.826937 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 22:48:53.827108 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.827020 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.827108 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.827053 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.827210 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.827140 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.827210 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.827152 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6qjt\"" Oct 31 22:48:53.829506 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.829483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.829506 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.829501 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.829689 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.829593 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pn67g\"" Oct 31 22:48:53.829689 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.829653 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.832686 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.832084 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.832686 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.832196 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gr9dz\"" Oct 31 22:48:53.832686 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.832229 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.832686 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.832302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.834599 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.834579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.835079 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.834983 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 22:48:53.835079 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.835042 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.835079 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.835056 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.835309 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.835276 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 22:48:53.835415 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.835327 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j2f6b\"" Oct 31 22:48:53.837372 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837352 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 22:48:53.837460 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837424 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 22:48:53.837460 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837430 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.837460 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.837632 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837501 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.837632 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.837509 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:48:53.837632 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837586 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 22:48:53.837764 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837671 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 22:48:53.837764 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.837686 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdsw9\"" Oct 31 22:48:53.839077 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.839061 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 22:48:53.841098 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.839805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.842080 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.842064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:53.842257 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.842234 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:48:53.844867 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.844843 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:53.846831 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.846811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:53.846901 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.846871 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-modprobe-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4t5z\" (UniqueName: \"kubernetes.io/projected/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-kube-api-access-b4t5z\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13e8e748-f088-43d1-8011-bf6c8f91ec59-hosts-file\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-socket-dir-parent\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-multus\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-conf\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-hostroot\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-bin\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-script-lib\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-registration-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-sys\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-host\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.850011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-dbus\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-systemd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-socket-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-kubelet\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.849981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-sys-fs\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-run\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-system-cni-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e8e748-f088-43d1-8011-bf6c8f91ec59-tmp-dir\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-bin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-netns\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-conf-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-etc-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cnibin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-daemon-config\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-var-lib-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.851277 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850640 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-netd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovn-node-metrics-cert\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysconfig\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-netns\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-multus-certs\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-kubelet\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.850996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-ovn\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-device-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-node-log\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-log-socket\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-config\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-env-overrides\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-var-lib-kubelet\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.853809 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-systemd-units\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwwt\" (UniqueName: \"kubernetes.io/projected/c8c42546-d756-4965-96c3-e3e05dd3d89b-kube-api-access-5nwwt\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnkd\" (UniqueName: \"kubernetes.io/projected/13e8e748-f088-43d1-8011-bf6c8f91ec59-kube-api-access-nsnkd\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-kubelet-config\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-system-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-etc-kubernetes\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gll\" (UniqueName: \"kubernetes.io/projected/f3c6a49a-2e5b-460c-8c6a-ca303019575a-kube-api-access-x5gll\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-systemd\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cni-binary-copy\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-k8s-cni-cncf-io\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-slash\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.851959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-kubernetes\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cnibin\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.854816 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-tuned\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2lq\" (UniqueName: \"kubernetes.io/projected/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-kube-api-access-dr2lq\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852238 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-lib-modules\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-tmp\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.855633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.852343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwpf\" (UniqueName: \"kubernetes.io/projected/8a1ca4ab-6864-49ea-af42-3c31f7823285-kube-api-access-plwpf\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.871645 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.871617 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p96sg" Oct 31 22:48:53.882965 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.882943 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p96sg" Oct 31 22:48:53.919840 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.919794 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"a70adfcc128189ec0c0f585313e4a644","Type":"ContainerStarted","Data":"25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894"} Oct 31 22:48:53.923067 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.923042 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"c87261ed136f2c9536dc41bf58fda7ff","Type":"ContainerStarted","Data":"efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9"} Oct 31 22:48:53.928020 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.927980 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 22:48:53.928160 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928146 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 22:48:53.928391 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928374 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkn4d\"" Oct 31 22:48:53.928489 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928429 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 22:48:53.928489 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928459 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.928608 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928503 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg7vs\"" Oct 31 22:48:53.928608 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928534 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 22:48:53.928608 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928587 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2gb2z\"" Oct 31 22:48:53.928744 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928628 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 22:48:53.928744 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928676 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2drsv\"" Oct 31 22:48:53.928744 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928701 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.928744 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928721 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 22:48:53.928916 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928756 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 22:48:53.928916 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.928775 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:48:53.940893 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.940870 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 22:48:53.953443 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cnibin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.953563 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-daemon-config\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.953563 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-var-lib-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953563 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-netd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953563 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953563 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953536 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cnibin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovn-node-metrics-cert\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953598 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysconfig\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953637 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-var-lib-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-netns\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-multus-certs\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-kubelet\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-ovn\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-device-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9779be5d-063f-4f06-8e58-746db277ce66-iptables-alerter-script\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9868\" (UniqueName: \"kubernetes.io/projected/37b0f64a-6235-4472-b7e2-a2d2a08fd388-kube-api-access-g9868\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-node-log\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.953871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-log-socket\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-config\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-env-overrides\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-var-lib-kubelet\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953954 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.953963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-systemd-units\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwwt\" (UniqueName: \"kubernetes.io/projected/c8c42546-d756-4965-96c3-e3e05dd3d89b-kube-api-access-5nwwt\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954032 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-daemon-config\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnkd\" (UniqueName: \"kubernetes.io/projected/13e8e748-f088-43d1-8011-bf6c8f91ec59-kube-api-access-nsnkd\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-kubelet-config\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-kubelet-config\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-netd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.954176 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-node-log\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.954222 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret podName:ce8955fd-a862-461a-bce9-a5be688358ef nodeName:}" failed. No retries permitted until 2026-10-31 22:48:54.454206315 +0000 UTC m=+3.037560024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret") pod "global-pull-secret-syncer-xrfs4" (UID: "ce8955fd-a862-461a-bce9-a5be688358ef") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-log-socket\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.954516 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954256 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-ovn\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-kubelet\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954265 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-var-lib-kubelet\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954291 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-device-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-netns\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysconfig\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-system-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-etc-kubernetes\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gll\" (UniqueName: \"kubernetes.io/projected/f3c6a49a-2e5b-460c-8c6a-ca303019575a-kube-api-access-x5gll\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-systemd\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcqf\" (UniqueName: \"kubernetes.io/projected/9779be5d-063f-4f06-8e58-746db277ce66-kube-api-access-vfcqf\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cni-binary-copy\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-k8s-cni-cncf-io\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-slash\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-kubernetes\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqdw\" (UniqueName: \"kubernetes.io/projected/6c707771-1798-4442-9277-cecbc002c007-kube-api-access-nxqdw\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:53.955188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cnibin\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-tuned\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37b0f64a-6235-4472-b7e2-a2d2a08fd388-serviceca\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2lq\" (UniqueName: \"kubernetes.io/projected/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-kube-api-access-dr2lq\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954913 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-multus-certs\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954924 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-env-overrides\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-lib-modules\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954968 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-systemd-units\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-tmp\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955006 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-slash\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwpf\" (UniqueName: \"kubernetes.io/projected/8a1ca4ab-6864-49ea-af42-3c31f7823285-kube-api-access-plwpf\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-modprobe-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.954876 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-config\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4t5z\" (UniqueName: \"kubernetes.io/projected/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-kube-api-access-b4t5z\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955194 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13e8e748-f088-43d1-8011-bf6c8f91ec59-hosts-file\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955243 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13e8e748-f088-43d1-8011-bf6c8f91ec59-hosts-file\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-socket-dir-parent\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955304 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-kubernetes\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955335 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-multus\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955344 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cnibin\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955249 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-multus\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-socket-dir-parent\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955412 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.956794 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-conf\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-hostroot\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-bin\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-script-lib\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-hostroot\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-registration-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-sys\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-host\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-dbus\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9779be5d-063f-4f06-8e58-746db277ce66-host-slash\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-systemd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-socket-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37b0f64a-6235-4472-b7e2-a2d2a08fd388-host\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-systemd\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-kubelet\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955900 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-sys-fs\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-run\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.957847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-k8s-cni-cncf-io\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-system-cni-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955983 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-script-lib\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e8e748-f088-43d1-8011-bf6c8f91ec59-tmp-dir\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956011 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-conf\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a940d2c-fc2c-40ae-a8bd-5916089eb925-agent-certs\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956047 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956089 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-dbus\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956092 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-system-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956118 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-lib-modules\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-etc-kubernetes\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-bin\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-systemd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-socket-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-kubelet\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-run\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956344 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cni-binary-copy\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.958780 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.955913 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-registration-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-system-cni-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a940d2c-fc2c-40ae-a8bd-5916089eb925-konnectivity-ca\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956428 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-sys-fs\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-bin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-bin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-netns\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-host\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-conf-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956623 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e8e748-f088-43d1-8011-bf6c8f91ec59-tmp-dir\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-etc-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956702 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-netns\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-etc-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956826 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-conf-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956940 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-modprobe-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.959390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956975 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.956989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-sys\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.959991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.957292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.959991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.958502 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovn-node-metrics-cert\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.959991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.959729 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-tuned\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.959991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.959757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-tmp\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:53.968981 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.968015 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:48:53.968981 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.968036 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:48:53.968981 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.968049 2576 projected.go:196] Error preparing data for projected volume kube-api-access-svcv2 for pod openshift-network-diagnostics/network-check-target-56mmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:53.968981 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:53.968101 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2 podName:7533bb1d-7252-4b45-b915-969a3f216ba4 nodeName:}" failed. No retries permitted until 2026-10-31 22:48:54.468084615 +0000 UTC m=+3.051438325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-svcv2" (UniqueName: "kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2") pod "network-check-target-56mmn" (UID: "7533bb1d-7252-4b45-b915-969a3f216ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:53.970852 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.970825 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gll\" (UniqueName: \"kubernetes.io/projected/f3c6a49a-2e5b-460c-8c6a-ca303019575a-kube-api-access-x5gll\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:48:53.972613 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.972592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2lq\" (UniqueName: \"kubernetes.io/projected/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-kube-api-access-dr2lq\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:53.972744 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.972719 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwwt\" (UniqueName: \"kubernetes.io/projected/c8c42546-d756-4965-96c3-e3e05dd3d89b-kube-api-access-5nwwt\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:53.973155 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.973125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnkd\" (UniqueName: \"kubernetes.io/projected/13e8e748-f088-43d1-8011-bf6c8f91ec59-kube-api-access-nsnkd\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:53.973466 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.973427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwpf\" (UniqueName: \"kubernetes.io/projected/8a1ca4ab-6864-49ea-af42-3c31f7823285-kube-api-access-plwpf\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:53.974243 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:53.974225 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4t5z\" (UniqueName: \"kubernetes.io/projected/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-kube-api-access-b4t5z\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:54.057181 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9779be5d-063f-4f06-8e58-746db277ce66-host-slash\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.057300 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37b0f64a-6235-4472-b7e2-a2d2a08fd388-host\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.057300 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a940d2c-fc2c-40ae-a8bd-5916089eb925-agent-certs\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:54.057300 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a940d2c-fc2c-40ae-a8bd-5916089eb925-konnectivity-ca\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:54.057300 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9779be5d-063f-4f06-8e58-746db277ce66-iptables-alerter-script\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.057488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9868\" (UniqueName: \"kubernetes.io/projected/37b0f64a-6235-4472-b7e2-a2d2a08fd388-kube-api-access-g9868\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.057488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcqf\" (UniqueName: \"kubernetes.io/projected/9779be5d-063f-4f06-8e58-746db277ce66-kube-api-access-vfcqf\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.057488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqdw\" (UniqueName: \"kubernetes.io/projected/6c707771-1798-4442-9277-cecbc002c007-kube-api-access-nxqdw\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:54.057488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37b0f64a-6235-4472-b7e2-a2d2a08fd388-serviceca\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.057488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:54.057683 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.057532 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:54.057683 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.057598 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:48:54.557580888 +0000 UTC m=+3.140934600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:54.057806 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057788 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9779be5d-063f-4f06-8e58-746db277ce66-host-slash\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.057844 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.057836 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37b0f64a-6235-4472-b7e2-a2d2a08fd388-host\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.058599 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.058574 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37b0f64a-6235-4472-b7e2-a2d2a08fd388-serviceca\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.058895 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.058862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a940d2c-fc2c-40ae-a8bd-5916089eb925-konnectivity-ca\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:54.060079 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.060057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9779be5d-063f-4f06-8e58-746db277ce66-iptables-alerter-script\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.060432 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.060410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a940d2c-fc2c-40ae-a8bd-5916089eb925-agent-certs\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:54.072300 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.072245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcqf\" (UniqueName: \"kubernetes.io/projected/9779be5d-063f-4f06-8e58-746db277ce66-kube-api-access-vfcqf\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.073779 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.073756 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqdw\" (UniqueName: \"kubernetes.io/projected/6c707771-1798-4442-9277-cecbc002c007-kube-api-access-nxqdw\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:54.074480 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.074457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9868\" (UniqueName: \"kubernetes.io/projected/37b0f64a-6235-4472-b7e2-a2d2a08fd388-kube-api-access-g9868\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.145981 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.145957 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:48:54.151052 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.151026 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:48:54.157921 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.157894 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c42546_d756_4965_96c3_e3e05dd3d89b.slice/crio-a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8 WatchSource:0}: Error finding container a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8: Status 404 returned error can't find the container with id a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8 Oct 31 22:48:54.160792 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.160738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sk99x" Oct 31 22:48:54.161067 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.161041 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d96fb7_909c_4e64_b5e8_8cdeccf7eca3.slice/crio-6ad96425a9201ac64d697ef4cf54be857254dbbf85e59a1b472edb6f0d01a103 WatchSource:0}: Error finding container 6ad96425a9201ac64d697ef4cf54be857254dbbf85e59a1b472edb6f0d01a103: Status 404 returned error can't find the container with id 6ad96425a9201ac64d697ef4cf54be857254dbbf85e59a1b472edb6f0d01a103 Oct 31 22:48:54.167994 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.167972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8d5v6" Oct 31 22:48:54.170373 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.170349 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e8e748_f088_43d1_8011_bf6c8f91ec59.slice/crio-3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663 WatchSource:0}: Error finding container 3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663: Status 404 returned error can't find the container with id 3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663 Oct 31 22:48:54.177412 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.177392 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c6a49a_2e5b_460c_8c6a_ca303019575a.slice/crio-bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f WatchSource:0}: Error finding container bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f: Status 404 returned error can't find the container with id bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f Oct 31 22:48:54.195294 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.195276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:48:54.206726 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.206703 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1ca4ab_6864_49ea_af42_3c31f7823285.slice/crio-001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641 WatchSource:0}: Error finding container 001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641: Status 404 returned error can't find the container with id 001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641 Oct 31 22:48:54.207633 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.207615 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:48:54.210442 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.210423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:48:54.217655 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.217634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:48:54.219703 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.219683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:48:54.220133 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.220108 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a940d2c_fc2c_40ae_a8bd_5916089eb925.slice/crio-1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89 WatchSource:0}: Error finding container 1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89: Status 404 returned error can't find the container with id 1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89 Oct 31 22:48:54.220865 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.220838 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9779be5d_063f_4f06_8e58_746db277ce66.slice/crio-ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba WatchSource:0}: Error finding container ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba: Status 404 returned error can't find the container with id ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba Oct 31 22:48:54.229564 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.229521 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffdc8b2_d568_4e04_a6a8_681de62c2ab5.slice/crio-d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b WatchSource:0}: Error finding container d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b: Status 404 returned error can't find the container with id d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b Oct 31 22:48:54.235103 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:48:54.235083 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b0f64a_6235_4472_b7e2_a2d2a08fd388.slice/crio-4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f WatchSource:0}: Error finding container 4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f: Status 404 returned error can't find the container with id 4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f Oct 31 22:48:54.252858 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.252796 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:48:54.460499 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.460352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:54.460675 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.460507 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:54.460675 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.460587 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret podName:ce8955fd-a862-461a-bce9-a5be688358ef nodeName:}" failed. No retries permitted until 2026-10-31 22:48:55.460566532 +0000 UTC m=+4.043920231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret") pod "global-pull-secret-syncer-xrfs4" (UID: "ce8955fd-a862-461a-bce9-a5be688358ef") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:54.561437 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.561402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:54.561610 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.561458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:54.561687 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.561614 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:48:54.561687 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.561631 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:48:54.561687 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.561640 2576 projected.go:196] Error preparing data for projected volume kube-api-access-svcv2 for pod openshift-network-diagnostics/network-check-target-56mmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:54.561835 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.561695 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2 podName:7533bb1d-7252-4b45-b915-969a3f216ba4 nodeName:}" failed. No retries permitted until 2026-10-31 22:48:55.561675527 +0000 UTC m=+4.145029221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-svcv2" (UniqueName: "kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2") pod "network-check-target-56mmn" (UID: "7533bb1d-7252-4b45-b915-969a3f216ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:54.562084 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.562064 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:54.562212 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:54.562107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:48:55.562096711 +0000 UTC m=+4.145450406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:54.889976 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.889841 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:43:53 +0000 UTC" deadline="2028-05-01 23:44:34.528064998 +0000 UTC" Oct 31 22:48:54.889976 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.889909 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13152h55m39.638160693s" Oct 31 22:48:54.929731 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.929695 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" event={"ID":"c8c42546-d756-4965-96c3-e3e05dd3d89b","Type":"ContainerStarted","Data":"a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8"} Oct 31 22:48:54.933502 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.933467 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58lh2" event={"ID":"37b0f64a-6235-4472-b7e2-a2d2a08fd388","Type":"ContainerStarted","Data":"4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f"} Oct 31 22:48:54.946460 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.946426 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerStarted","Data":"d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b"} Oct 31 22:48:54.956309 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.956277 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q85ln" event={"ID":"9779be5d-063f-4f06-8e58-746db277ce66","Type":"ContainerStarted","Data":"ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba"} Oct 31 22:48:54.961774 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.961650 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpmrk" event={"ID":"5a940d2c-fc2c-40ae-a8bd-5916089eb925","Type":"ContainerStarted","Data":"1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89"} Oct 31 22:48:54.963658 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.963633 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641"} Oct 31 22:48:54.967101 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.967076 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8d5v6" event={"ID":"f3c6a49a-2e5b-460c-8c6a-ca303019575a","Type":"ContainerStarted","Data":"bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f"} Oct 31 22:48:54.973557 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.973515 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gljgz" event={"ID":"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3","Type":"ContainerStarted","Data":"6ad96425a9201ac64d697ef4cf54be857254dbbf85e59a1b472edb6f0d01a103"} Oct 31 22:48:54.979286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:54.979217 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sk99x" event={"ID":"13e8e748-f088-43d1-8011-bf6c8f91ec59","Type":"ContainerStarted","Data":"3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663"} Oct 31 22:48:55.468798 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.468767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:55.468990 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.468973 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:55.469068 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.469043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret podName:ce8955fd-a862-461a-bce9-a5be688358ef nodeName:}" failed. No retries permitted until 2026-10-31 22:48:57.469025006 +0000 UTC m=+6.052378717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret") pod "global-pull-secret-syncer-xrfs4" (UID: "ce8955fd-a862-461a-bce9-a5be688358ef") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:55.569390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.569158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:55.569524 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.569420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:55.569603 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.569578 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:48:55.569603 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.569599 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:48:55.569709 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.569612 2576 projected.go:196] Error preparing data for projected volume kube-api-access-svcv2 for pod openshift-network-diagnostics/network-check-target-56mmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:55.569709 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.569667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2 podName:7533bb1d-7252-4b45-b915-969a3f216ba4 nodeName:}" failed. No retries permitted until 2026-10-31 22:48:57.56964779 +0000 UTC m=+6.153001498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-svcv2" (UniqueName: "kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2") pod "network-check-target-56mmn" (UID: "7533bb1d-7252-4b45-b915-969a3f216ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:55.570181 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.570050 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:55.570181 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.570111 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:48:57.570089385 +0000 UTC m=+6.153443080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:55.631122 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.631064 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:48:55.890481 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.890348 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:43:53 +0000 UTC" deadline="2028-06-21 05:47:07.19109251 +0000 UTC" Oct 31 22:48:55.890481 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.890411 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14358h58m11.300685445s" Oct 31 22:48:55.916914 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.916887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:55.917080 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.917012 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:48:55.917433 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.917414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:55.917534 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.917520 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:48:55.917617 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:55.917597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:55.917733 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:55.917669 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:48:56.002211 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:56.002133 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"a70adfcc128189ec0c0f585313e4a644","Type":"ContainerStarted","Data":"7b3ac75c44746ef6472efec9f018886048d7548d589b4dd94772ae3a2f1401a2"} Oct 31 22:48:56.007490 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:56.007459 2576 generic.go:356] "Generic (PLEG): container finished" podID="c87261ed136f2c9536dc41bf58fda7ff" containerID="18a893e7134d7ffb74b9587b0932c2d49f340649a1a3aa252f0eb29b4a867928" exitCode=0 Oct 31 22:48:56.007761 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:56.007506 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"c87261ed136f2c9536dc41bf58fda7ff","Type":"ContainerDied","Data":"18a893e7134d7ffb74b9587b0932c2d49f340649a1a3aa252f0eb29b4a867928"} Oct 31 22:48:56.035084 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:56.035014 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" podStartSLOduration=3.035001587 podStartE2EDuration="3.035001587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:48:56.018764146 +0000 UTC m=+4.602117863" watchObservedRunningTime="2026-10-31 22:48:56.035001587 +0000 UTC m=+4.618355306" Oct 31 22:48:57.021825 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.021791 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"c87261ed136f2c9536dc41bf58fda7ff","Type":"ContainerStarted","Data":"cfe67ac2594be1a31066cc11d050ce0ed91aa3e517cf4d6b6a9cec53f4de0f38"} Oct 31 22:48:57.037246 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.037155 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" podStartSLOduration=4.037139237 podStartE2EDuration="4.037139237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:48:57.036782366 +0000 UTC m=+5.620136083" watchObservedRunningTime="2026-10-31 22:48:57.037139237 +0000 UTC m=+5.620492955" Oct 31 22:48:57.490731 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.490698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:57.490913 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.490899 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:57.490969 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.490958 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret podName:ce8955fd-a862-461a-bce9-a5be688358ef nodeName:}" failed. No retries permitted until 2026-10-31 22:49:01.490942333 +0000 UTC m=+10.074296044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret") pod "global-pull-secret-syncer-xrfs4" (UID: "ce8955fd-a862-461a-bce9-a5be688358ef") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:48:57.592052 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.591995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:57.592052 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.592043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:57.592286 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.592191 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:48:57.592286 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.592213 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:48:57.592286 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.592226 2576 projected.go:196] Error preparing data for projected volume kube-api-access-svcv2 for pod openshift-network-diagnostics/network-check-target-56mmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:57.592286 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.592281 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2 podName:7533bb1d-7252-4b45-b915-969a3f216ba4 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:01.592263693 +0000 UTC m=+10.175617392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-svcv2" (UniqueName: "kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2") pod "network-check-target-56mmn" (UID: "7533bb1d-7252-4b45-b915-969a3f216ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:48:57.592783 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.592761 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:57.592873 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.592817 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:01.592802116 +0000 UTC m=+10.176155814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:48:57.914295 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.914265 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:57.914468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.914400 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:48:57.914860 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.914837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:57.914982 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.914915 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:48:57.914982 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:57.914951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:57.915106 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:57.914995 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:48:58.029234 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:58.029177 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="60a91d0230f9fd80b277dbf00d216adefa76b505abfe57d2a7b23dc96ba343f7" exitCode=0 Oct 31 22:48:58.030264 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:58.030237 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerDied","Data":"60a91d0230f9fd80b277dbf00d216adefa76b505abfe57d2a7b23dc96ba343f7"} Oct 31 22:48:59.038341 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:59.038303 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpmrk" event={"ID":"5a940d2c-fc2c-40ae-a8bd-5916089eb925","Type":"ContainerStarted","Data":"c4d6468a8a24f7f387661f14b20f859892790f15f257c40865ba1902a9306b85"} Oct 31 22:48:59.914841 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:59.914781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:48:59.915006 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:59.914908 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:48:59.915315 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:59.915294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:48:59.915418 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:59.915397 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:48:59.915481 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:48:59.915469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:48:59.915541 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:48:59.915526 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:00.041902 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:00.041872 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58lh2" event={"ID":"37b0f64a-6235-4472-b7e2-a2d2a08fd388","Type":"ContainerStarted","Data":"26caa58248fc7adf0d38b4e23f96a884032872f2f35a770e2a74469948505eea"} Oct 31 22:49:00.045505 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:00.045478 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" event={"ID":"c8c42546-d756-4965-96c3-e3e05dd3d89b","Type":"ContainerStarted","Data":"e0bacb8fcf4bfc421543a85e4f60c9b4faecf99a9cc250b49b2f7f2455c8c902"} Oct 31 22:49:00.059495 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:00.059418 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jpmrk" podStartSLOduration=4.064422412 podStartE2EDuration="8.059400449s" totalImagesPullingTime="3.994978037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:48:59.054740479 +0000 UTC m=+7.638094207" watchObservedRunningTime="2026-10-31 22:49:00.059400449 +0000 UTC m=+8.642754167" Oct 31 22:49:00.060162 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:00.060122 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-58lh2" podStartSLOduration=3.120266355 podStartE2EDuration="8.060111694s" totalImagesPullingTime="4.939845339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:00.059085196 +0000 UTC m=+8.642438912" watchObservedRunningTime="2026-10-31 22:49:00.060111694 +0000 UTC m=+8.643465410" Oct 31 22:49:01.522818 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.522710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:01.523215 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.522875 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:01.523215 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.522952 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret podName:ce8955fd-a862-461a-bce9-a5be688358ef nodeName:}" failed. No retries permitted until 2026-10-31 22:49:09.522932186 +0000 UTC m=+18.106285894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret") pod "global-pull-secret-syncer-xrfs4" (UID: "ce8955fd-a862-461a-bce9-a5be688358ef") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:01.572174 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.572101 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 22:49:01.630574 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.630436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:01.630574 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.630474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:01.630788 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.630592 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:01.630788 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.630612 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:01.630788 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.630630 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:01.630788 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.630641 2576 projected.go:196] Error preparing data for projected volume kube-api-access-svcv2 for pod openshift-network-diagnostics/network-check-target-56mmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:01.630788 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.630655 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:09.630637155 +0000 UTC m=+18.213990849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:01.630788 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.630690 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2 podName:7533bb1d-7252-4b45-b915-969a3f216ba4 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:09.63067261 +0000 UTC m=+18.214026327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-svcv2" (UniqueName: "kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2") pod "network-check-target-56mmn" (UID: "7533bb1d-7252-4b45-b915-969a3f216ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:01.915457 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.915386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:01.915629 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.915496 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:01.916065 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.915916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:01.916065 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.916024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:01.916340 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.916313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:01.916425 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:01.916411 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:01.919329 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.918664 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T22:49:01.572116665Z","UUID":"96f9c3fc-183b-48ba-9b50-1a63e7ab3339","Handler":null,"Name":"","Endpoint":""} Oct 31 22:49:01.920574 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.920218 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 22:49:01.920574 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:01.920243 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 22:49:02.052099 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:02.052001 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" event={"ID":"c8c42546-d756-4965-96c3-e3e05dd3d89b","Type":"ContainerStarted","Data":"bba4ca0c0df98dd54edf3f3dff590d7f5a796b63c446f1827c92c57c1ac2accf"} Oct 31 22:49:02.266689 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:02.266615 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:02.267132 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:02.267105 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:03.056998 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.056963 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="647a55ba56e9e838475c9b0d014f61c9f50ab3170e930657739524b9437c1d94" exitCode=0 Oct 31 22:49:03.057478 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.057052 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerDied","Data":"647a55ba56e9e838475c9b0d014f61c9f50ab3170e930657739524b9437c1d94"} Oct 31 22:49:03.059939 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.059773 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8d5v6" event={"ID":"f3c6a49a-2e5b-460c-8c6a-ca303019575a","Type":"ContainerStarted","Data":"6703181c237d0ac7751c9f44e19dea1a946a288451363401c6527f9f226337b6"} Oct 31 22:49:03.062695 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.062645 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sk99x" event={"ID":"13e8e748-f088-43d1-8011-bf6c8f91ec59","Type":"ContainerStarted","Data":"f2a056aa038327c9d6d18c3f74927e49f1d577996c21534715c944cc725055aa"} Oct 31 22:49:03.065870 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.065843 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" event={"ID":"c8c42546-d756-4965-96c3-e3e05dd3d89b","Type":"ContainerStarted","Data":"351f50ff292f4930536a8c7838711bf74f6665ab5718832846cb5891556e2c4d"} Oct 31 22:49:03.095336 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.095255 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sk99x" podStartSLOduration=3.317026505 podStartE2EDuration="11.095242322s" totalImagesPullingTime="7.778215817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:03.095242785 +0000 UTC m=+11.678596502" watchObservedRunningTime="2026-10-31 22:49:03.095242322 +0000 UTC m=+11.678596038" Oct 31 22:49:03.112600 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.112541 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podStartSLOduration=2.806676697 podStartE2EDuration="11.112529802s" totalImagesPullingTime="8.305853105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:03.112055432 +0000 UTC m=+11.695409150" watchObservedRunningTime="2026-10-31 22:49:03.112529802 +0000 UTC m=+11.695883518" Oct 31 22:49:03.918724 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.918486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:03.918851 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:03.918827 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:03.919219 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.919179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:03.919292 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:03.919274 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:03.919353 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:03.919326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:03.919402 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:03.919387 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:04.070101 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:04.070070 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerStarted","Data":"4fc9f98684fce4cfba96e3236dea3e2613d9faf4c092ef13d546adfd1d74c99b"} Oct 31 22:49:04.071808 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:04.071780 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q85ln" event={"ID":"9779be5d-063f-4f06-8e58-746db277ce66","Type":"ContainerStarted","Data":"61baa0cf82c5866915f76311c3847cabb42ba82d17c7e4ed7e1baaeb1ab4a323"} Oct 31 22:49:04.092946 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:04.092900 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8d5v6" podStartSLOduration=4.250205295 podStartE2EDuration="12.092883704s" totalImagesPullingTime="7.842678409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:03.129314909 +0000 UTC m=+11.712668625" watchObservedRunningTime="2026-10-31 22:49:04.092883704 +0000 UTC m=+12.676237421" Oct 31 22:49:04.110216 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:04.110170 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q85ln" podStartSLOduration=4.312271277 podStartE2EDuration="12.11015329s" totalImagesPullingTime="7.797882013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:04.109750331 +0000 UTC m=+12.693104048" watchObservedRunningTime="2026-10-31 22:49:04.11015329 +0000 UTC m=+12.693507006" Oct 31 22:49:05.076102 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.076071 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="4fc9f98684fce4cfba96e3236dea3e2613d9faf4c092ef13d546adfd1d74c99b" exitCode=0 Oct 31 22:49:05.076707 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.076160 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerDied","Data":"4fc9f98684fce4cfba96e3236dea3e2613d9faf4c092ef13d546adfd1d74c99b"} Oct 31 22:49:05.077871 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.077844 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gljgz" event={"ID":"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3","Type":"ContainerStarted","Data":"9e1f0ce09673a76694327d72c17c4376ff6fcaf5d905dcf870bcca14a37543dc"} Oct 31 22:49:05.118111 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.118072 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gljgz" podStartSLOduration=2.956582015 podStartE2EDuration="13.11805743s" totalImagesPullingTime="10.161475415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:05.117614004 +0000 UTC m=+13.700967719" watchObservedRunningTime="2026-10-31 22:49:05.11805743 +0000 UTC m=+13.701411146" Oct 31 22:49:05.918011 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.917983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:05.918151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.917983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:05.918151 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:05.918124 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:05.918256 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:05.918195 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:05.918256 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:05.917998 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:05.918352 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:05.918305 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:06.083406 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:06.083374 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="c06a7f6811c3e37774521e5dd901bd210df6c0731713ae210c2e660be65fc2cb" exitCode=0 Oct 31 22:49:06.084062 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:06.083466 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerDied","Data":"c06a7f6811c3e37774521e5dd901bd210df6c0731713ae210c2e660be65fc2cb"} Oct 31 22:49:07.915205 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:07.915167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:07.915684 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:07.915218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:07.915684 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:07.915167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:07.915684 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:07.915317 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:07.915684 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:07.915420 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:07.915684 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:07.915533 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:09.092831 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.092646 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="7454988180ec6e403d3847ad2bb0d4d2b1e628fd4f035333755ba46b139bbb16" exitCode=0 Oct 31 22:49:09.092831 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.092726 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerDied","Data":"7454988180ec6e403d3847ad2bb0d4d2b1e628fd4f035333755ba46b139bbb16"} Oct 31 22:49:09.591159 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.590978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:09.591292 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.591113 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:09.591292 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.591248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret podName:ce8955fd-a862-461a-bce9-a5be688358ef nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.591233359 +0000 UTC m=+34.174587055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret") pod "global-pull-secret-syncer-xrfs4" (UID: "ce8955fd-a862-461a-bce9-a5be688358ef") : object "kube-system"/"original-pull-secret" not registered Oct 31 22:49:09.691779 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.691710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:09.691779 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.691767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:09.691940 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.691835 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:09.691940 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.691840 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 22:49:09.691940 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.691855 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 22:49:09.691940 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.691864 2576 projected.go:196] Error preparing data for projected volume kube-api-access-svcv2 for pod openshift-network-diagnostics/network-check-target-56mmn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:09.691940 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.691878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.691867815 +0000 UTC m=+34.275221509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 22:49:09.691940 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.691902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2 podName:7533bb1d-7252-4b45-b915-969a3f216ba4 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.691892526 +0000 UTC m=+34.275246221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-svcv2" (UniqueName: "kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2") pod "network-check-target-56mmn" (UID: "7533bb1d-7252-4b45-b915-969a3f216ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 22:49:09.914900 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.914867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:09.914900 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.914891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:09.914900 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:09.914906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:09.915141 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.914983 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:09.915141 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.915041 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:09.915141 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:09.915121 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:10.097597 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.097523 2576 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="7a7580bf8f4de6f8b2edc4479db5f361d3a272f910499c7357cfa25232062a74" exitCode=0 Oct 31 22:49:10.098013 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.097617 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerDied","Data":"7a7580bf8f4de6f8b2edc4479db5f361d3a272f910499c7357cfa25232062a74"} Oct 31 22:49:10.100820 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.100799 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"1a6da02c03d54b14c12fdc822442d51c6a2701a94eee99a14e9ff00e2fd0a25e"} Oct 31 22:49:10.100887 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.100854 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"b262013aa10d0f6cff00ccdcc1e6613d8ed7d104cca155730363d26ccf8988d3"} Oct 31 22:49:10.100887 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.100884 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"6d26fa864f5e889063933185523f9c97766f4be2245b377fa1aafeaa0df670ab"} Oct 31 22:49:10.100941 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.100911 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"963435ae11d5ceb9b27731690a2cb21fb0347301a9c6235bde0d9d53b83fa4d8"} Oct 31 22:49:10.100941 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.100938 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"e4851e5e3acfda14abb29b113ce968f32bbf35dcc96003bd0cc58541c4e3f6e0"} Oct 31 22:49:10.101007 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:10.100965 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"7bd5acbfb022fc1b3ea2a613f0aaeff524b1cca5a453f89083d995bd515c24d9"} Oct 31 22:49:11.106109 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:11.106077 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" event={"ID":"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5","Type":"ContainerStarted","Data":"59b572d2a2fc7fbadcdefac0e917f3660fca43616489cc6aa4b8a89096426d33"} Oct 31 22:49:11.129390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:11.129348 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gbvzb" podStartSLOduration=7.91860371 podStartE2EDuration="19.129335937s" totalImagesPullingTime="11.210732227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:11.128844223 +0000 UTC m=+19.712197941" watchObservedRunningTime="2026-10-31 22:49:11.129335937 +0000 UTC m=+19.712689652" Oct 31 22:49:11.915114 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:11.915022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:11.915314 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:11.915117 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:11.915314 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:11.915142 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:11.915314 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:11.915186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:11.915314 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:11.915255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:11.915314 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:11.915185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:12.110629 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:12.110596 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"b293af67ce779179ef82870fb8f023739a979ab0707c3bf4a2c1838069cf66d2"} Oct 31 22:49:13.914828 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:13.914750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:13.914828 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:13.914778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:13.915506 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:13.914845 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:13.915506 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:13.914911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:13.915506 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:13.915020 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:13.915506 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:13.915140 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:14.118259 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:14.118224 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" event={"ID":"8a1ca4ab-6864-49ea-af42-3c31f7823285","Type":"ContainerStarted","Data":"3cda48ccb237335a990ed56a01f471d3628adba7af0d953d148b021cdf8402de"} Oct 31 22:49:14.118573 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:14.118532 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:14.118690 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:14.118627 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:14.133519 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:14.133496 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:14.148972 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:14.148934 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podStartSLOduration=7.260204744 podStartE2EDuration="22.148924783s" totalImagesPullingTime="14.888720039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:14.148741908 +0000 UTC m=+22.732095624" watchObservedRunningTime="2026-10-31 22:49:14.148924783 +0000 UTC m=+22.732278500" Oct 31 22:49:15.120636 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.120608 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:15.135084 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.135059 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:15.289708 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.289680 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xrfs4"] Oct 31 22:49:15.289834 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.289784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:15.289902 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:15.289858 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:15.293694 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.293666 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cz8ch"] Oct 31 22:49:15.293815 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.293732 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56mmn"] Oct 31 22:49:15.293815 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.293809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:15.293918 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:15.293896 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:15.293985 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:15.293970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:15.294084 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:15.294065 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:16.914693 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:16.914661 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:16.915142 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:16.914749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:16.915142 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:16.914754 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:16.915142 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:16.914822 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:16.915142 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:16.914834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:16.915142 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:16.914918 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:18.914870 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:18.914658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:18.915248 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:18.914719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:18.915248 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:18.914950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:18.915248 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:18.914727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:18.915248 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:18.915010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:18.915248 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:18.915101 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:20.914834 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:20.914813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:20.915220 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:20.914901 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56mmn" podUID="7533bb1d-7252-4b45-b915-969a3f216ba4" Oct 31 22:49:20.915220 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:20.914910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:20.915220 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:20.915010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cz8ch" podUID="6c707771-1798-4442-9277-cecbc002c007" Oct 31 22:49:20.915220 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:20.915032 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:20.915220 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:20.915083 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xrfs4" podUID="ce8955fd-a862-461a-bce9-a5be688358ef" Oct 31 22:49:21.724035 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.724008 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeReady" Oct 31 22:49:21.724182 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.724127 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 22:49:21.787584 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.787558 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k5t2f"] Oct 31 22:49:21.790535 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.790519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:21.793090 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.793065 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-khbjk"] Oct 31 22:49:21.795889 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.795870 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gftjj"] Oct 31 22:49:21.796007 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.795985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:21.801226 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.801197 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.801341 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.801327 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.804695 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.804676 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w"] Oct 31 22:49:21.804797 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.804744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:21.807414 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.807396 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9"] Oct 31 22:49:21.807531 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.807514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:21.810255 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.810240 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fj2lt"] Oct 31 22:49:21.810344 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.810331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:21.812956 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.812937 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8f9bc98f7-xn5rp"] Oct 31 22:49:21.813131 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.813112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.816373 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.816334 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688"] Oct 31 22:49:21.816819 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.816786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:21.820846 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.820818 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l"] Oct 31 22:49:21.820988 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.820974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:21.823958 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.823941 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vkldg"] Oct 31 22:49:21.824084 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.824067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:21.827028 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.827013 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-786cddd9dc-dfvvd"] Oct 31 22:49:21.827134 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.827120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:21.829629 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.829613 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9"] Oct 31 22:49:21.829736 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.829688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.832429 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.832411 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qv9l7"] Oct 31 22:49:21.832847 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.832825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:21.840674 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.836625 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r2hnp"] Oct 31 22:49:21.840674 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.837721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:21.843872 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.843853 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c6987c8-t24sb"] Oct 31 22:49:21.845579 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.844081 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 22:49:21.845579 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.844159 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 22:49:21.845579 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.844214 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 22:49:21.845579 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.844265 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 22:49:21.845579 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.844470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:21.845579 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.843989 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 22:49:21.847211 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.847196 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xctjg"] Oct 31 22:49:21.847359 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.847342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.851351 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.850382 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw"] Oct 31 22:49:21.864714 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.864694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:21.865501 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865476 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k5t2f"] Oct 31 22:49:21.865642 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865523 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-khbjk"] Oct 31 22:49:21.865642 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865562 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9"] Oct 31 22:49:21.865642 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865584 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gftjj"] Oct 31 22:49:21.865642 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865606 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w"] Oct 31 22:49:21.865642 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865626 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qv9l7"] Oct 31 22:49:21.865642 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865644 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865659 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vkldg"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865672 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865687 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2hnp"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865700 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8f9bc98f7-xn5rp"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865713 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865730 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fj2lt"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865748 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865765 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xctjg"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865785 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6987c8-t24sb"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865798 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-786cddd9dc-dfvvd"] Oct 31 22:49:21.865935 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.865853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:21.883476 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.883457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfklt\" (UniqueName: \"kubernetes.io/projected/e4043168-4d17-4609-bc2f-37951b7480b4-kube-api-access-kfklt\") pod \"kas-connection-checker-685895c7c9-k5t2f\" (UID: \"e4043168-4d17-4609-bc2f-37951b7480b4\") " pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:21.883580 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.883564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjjq\" (UniqueName: \"kubernetes.io/projected/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e-kube-api-access-4kjjq\") pod \"kas-connection-checker-685895c7c9-khbjk\" (UID: \"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e\") " pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:21.898864 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.898839 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.898965 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.898873 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.898965 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.898952 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899074 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.898990 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899074 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899000 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899086 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899108 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899126 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899147 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899154 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899128 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899186 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899156 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899240 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899248 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899296 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899314 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899343 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899356 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899318 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899373 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899385 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899411 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899424 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899439 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 22:49:21.899488 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899437 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 22:49:21.900144 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899538 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.900144 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.899748 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:21.903029 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.902369 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 22:49:21.903267 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.903238 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 22:49:21.903528 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.903514 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 22:49:21.903611 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.903595 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 22:49:21.939319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939271 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 22:49:21.939319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939286 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 22:49:21.939319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939272 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 22:49:21.939319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939302 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 22:49:21.939319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939297 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 22:49:21.939319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939286 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939446 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939454 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939492 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s2g2v\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939522 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fzlj2\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939595 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7g25v\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939682 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939707 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 22:49:21.940223 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.939771 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6jkj5\"" Oct 31 22:49:21.940739 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.940716 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hdfv2\"" Oct 31 22:49:21.958431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.958409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:21.984460 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-tmp\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.984610 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-default-certificate\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:21.984610 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvv7\" (UniqueName: \"kubernetes.io/projected/8dfd0ee8-e42a-4bbd-b355-44262861da53-kube-api-access-dcvv7\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:21.984610 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.984748 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjd2s\" (UniqueName: \"kubernetes.io/projected/7afda5df-8528-4304-aa1d-156aefde0156-kube-api-access-hjd2s\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:21.984748 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.984748 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:21.984748 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01689403-7698-4f38-b4c7-8a3603742ca8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:21.984748 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-service-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.984936 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-serving-cert\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:21.984936 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:21.984936 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.984936 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjjq\" (UniqueName: \"kubernetes.io/projected/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e-kube-api-access-4kjjq\") pod \"kas-connection-checker-685895c7c9-khbjk\" (UID: \"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e\") " pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:21.984936 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.984886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.985207 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eec52f11-3785-48d2-92a1-cbd30a6047e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:21.985207 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01689403-7698-4f38-b4c7-8a3603742ca8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:21.985207 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.985207 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:21.985207 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b06c9464-bdf9-493a-8d44-37cca42409d1-config\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfklt\" (UniqueName: \"kubernetes.io/projected/e4043168-4d17-4609-bc2f-37951b7480b4-kube-api-access-kfklt\") pod \"kas-connection-checker-685895c7c9-k5t2f\" (UID: \"e4043168-4d17-4609-bc2f-37951b7480b4\") " pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmq4\" (UniqueName: \"kubernetes.io/projected/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9-kube-api-access-2kmq4\") pod \"volume-data-source-validator-58c9f5f46d-q7688\" (UID: \"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dfd0ee8-e42a-4bbd-b355-44262861da53-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-cert\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4lb\" (UniqueName: \"kubernetes.io/projected/67c4f288-6571-456f-8aa8-d6f5947eb14d-kube-api-access-5c4lb\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:21.985398 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqq7\" (UniqueName: \"kubernetes.io/projected/aad03e4e-7139-463f-baf7-752544c6763f-kube-api-access-bvqq7\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b06c9464-bdf9-493a-8d44-37cca42409d1-serving-cert\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c4f288-6571-456f-8aa8-d6f5947eb14d-tmp-dir\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad03e4e-7139-463f-baf7-752544c6763f-serving-cert\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgzm\" (UniqueName: \"kubernetes.io/projected/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-kube-api-access-nvgzm\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8jn\" (UniqueName: \"kubernetes.io/projected/b06c9464-bdf9-493a-8d44-37cca42409d1-kube-api-access-6k8jn\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67c4f288-6571-456f-8aa8-d6f5947eb14d-config-volume\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:21.985778 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-snapshots\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fsgl\" (UniqueName: \"kubernetes.io/projected/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-kube-api-access-7fsgl\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-config\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrjl\" (UniqueName: \"kubernetes.io/projected/90d658ad-d248-4853-8bd8-b64076ec9968-kube-api-access-mwrjl\") pod \"network-check-source-6f78d59f99-cbfsw\" (UID: \"90d658ad-d248-4853-8bd8-b64076ec9968\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.985968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-stats-auth\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bjc\" (UniqueName: \"kubernetes.io/projected/3ce54792-3b37-4895-a39a-d233a7e3d4ee-kube-api-access-q4bjc\") pod \"kas-connection-checker-685895c7c9-gftjj\" (UID: \"3ce54792-3b37-4895-a39a-d233a7e3d4ee\") " pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlrr\" (UniqueName: \"kubernetes.io/projected/01689403-7698-4f38-b4c7-8a3603742ca8-kube-api-access-gdlrr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9qc\" (UniqueName: \"kubernetes.io/projected/b8edf410-4aa2-42ea-bf3f-016002b74ffa-kube-api-access-8t9qc\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:21.986286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:21.986181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.033381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.033354 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjjq\" (UniqueName: \"kubernetes.io/projected/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e-kube-api-access-4kjjq\") pod \"kas-connection-checker-685895c7c9-khbjk\" (UID: \"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e\") " pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:22.033485 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.033400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfklt\" (UniqueName: \"kubernetes.io/projected/e4043168-4d17-4609-bc2f-37951b7480b4-kube-api-access-kfklt\") pod \"kas-connection-checker-685895c7c9-k5t2f\" (UID: \"e4043168-4d17-4609-bc2f-37951b7480b4\") " pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:22.087063 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.087063 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087064 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9qc\" (UniqueName: \"kubernetes.io/projected/b8edf410-4aa2-42ea-bf3f-016002b74ffa-kube-api-access-8t9qc\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:22.087241 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.087241 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-tmp\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.087376 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-tmp\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.087437 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-default-certificate\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.087515 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvv7\" (UniqueName: \"kubernetes.io/projected/8dfd0ee8-e42a-4bbd-b355-44262861da53-kube-api-access-dcvv7\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:22.087606 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.087674 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjd2s\" (UniqueName: \"kubernetes.io/projected/7afda5df-8528-4304-aa1d-156aefde0156-kube-api-access-hjd2s\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.087674 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.087806 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:22.087806 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01689403-7698-4f38-b4c7-8a3603742ca8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.087806 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.087756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-service-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-serving-cert\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088032 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.088196 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eec52f11-3785-48d2-92a1-cbd30a6047e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01689403-7698-4f38-b4c7-8a3603742ca8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.088260 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert podName:eec52f11-3785-48d2-92a1-cbd30a6047e2 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.58824098 +0000 UTC m=+31.171594688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qv9l7" (UID: "eec52f11-3785-48d2-92a1-cbd30a6047e2") : secret "networking-console-plugin-cert" not found Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-service-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.088616 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.088425 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01689403-7698-4f38-b4c7-8a3603742ca8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.089336 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.089336 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eec52f11-3785-48d2-92a1-cbd30a6047e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:22.089468 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.089533 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.089500 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 22:49:22.089533 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.089650 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.089570 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls podName:67c4f288-6571-456f-8aa8-d6f5947eb14d nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.589533775 +0000 UTC m=+31.172887471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls") pod "dns-default-xctjg" (UID: "67c4f288-6571-456f-8aa8-d6f5947eb14d") : secret "dns-default-metrics-tls" not found Oct 31 22:49:22.089709 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:22.089757 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.089757 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b06c9464-bdf9-493a-8d44-37cca42409d1-config\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.089864 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmq4\" (UniqueName: \"kubernetes.io/projected/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9-kube-api-access-2kmq4\") pod \"volume-data-source-validator-58c9f5f46d-q7688\" (UID: \"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:22.089864 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dfd0ee8-e42a-4bbd-b355-44262861da53-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:22.089864 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-cert\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:22.089864 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4lb\" (UniqueName: \"kubernetes.io/projected/67c4f288-6571-456f-8aa8-d6f5947eb14d-kube-api-access-5c4lb\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqq7\" (UniqueName: \"kubernetes.io/projected/aad03e4e-7139-463f-baf7-752544c6763f-kube-api-access-bvqq7\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b06c9464-bdf9-493a-8d44-37cca42409d1-serving-cert\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c4f288-6571-456f-8aa8-d6f5947eb14d-tmp-dir\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.089975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad03e4e-7139-463f-baf7-752544c6763f-serving-cert\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.090124 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgzm\" (UniqueName: \"kubernetes.io/projected/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-kube-api-access-nvgzm\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8jn\" (UniqueName: \"kubernetes.io/projected/b06c9464-bdf9-493a-8d44-37cca42409d1-kube-api-access-6k8jn\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67c4f288-6571-456f-8aa8-d6f5947eb14d-config-volume\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-default-certificate\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-snapshots\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fsgl\" (UniqueName: \"kubernetes.io/projected/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-kube-api-access-7fsgl\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-config\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrjl\" (UniqueName: \"kubernetes.io/projected/90d658ad-d248-4853-8bd8-b64076ec9968-kube-api-access-mwrjl\") pod \"network-check-source-6f78d59f99-cbfsw\" (UID: \"90d658ad-d248-4853-8bd8-b64076ec9968\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090480 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-stats-auth\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.090634 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bjc\" (UniqueName: \"kubernetes.io/projected/3ce54792-3b37-4895-a39a-d233a7e3d4ee-kube-api-access-q4bjc\") pod \"kas-connection-checker-685895c7c9-gftjj\" (UID: \"3ce54792-3b37-4895-a39a-d233a7e3d4ee\") " pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlrr\" (UniqueName: \"kubernetes.io/projected/01689403-7698-4f38-b4c7-8a3603742ca8-kube-api-access-gdlrr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090610 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-serving-cert\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.090653 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.090351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.090749 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.090795 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.590777005 +0000 UTC m=+31.174130715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : secret "router-metrics-certs-default" not found Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.091172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.091203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dfd0ee8-e42a-4bbd-b355-44262861da53-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:22.091496 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.091414 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.092111 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.091582 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.092111 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.091620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.092111 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.091720 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:22.092111 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.091737 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:22.092111 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.091824 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.591808725 +0000 UTC m=+31.175162424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:22.092379 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.092316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-config\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.092834 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.092814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.094270 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.093160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01689403-7698-4f38-b4c7-8a3603742ca8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.094270 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.093350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b06c9464-bdf9-493a-8d44-37cca42409d1-config\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.093412 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.093454 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.093586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.593568042 +0000 UTC m=+31.176921759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.093586 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad03e4e-7139-463f-baf7-752544c6763f-serving-cert\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.093636 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.094329 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.59431298 +0000 UTC m=+31.177666886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.094344 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.094184 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.094115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b06c9464-bdf9-493a-8d44-37cca42409d1-serving-cert\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.094217 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67c4f288-6571-456f-8aa8-d6f5947eb14d-config-volume\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.094298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-snapshots\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.093914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c4f288-6571-456f-8aa8-d6f5947eb14d-tmp-dir\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.094396 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.594379621 +0000 UTC m=+31.177733316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:22.094431 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.094415 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls podName:b8edf410-4aa2-42ea-bf3f-016002b74ffa nodeName:}" failed. No retries permitted until 2026-10-31 22:49:22.594405743 +0000 UTC m=+31.177759444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5blk9" (UID: "b8edf410-4aa2-42ea-bf3f-016002b74ffa") : secret "samples-operator-tls" not found Oct 31 22:49:22.094929 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.094538 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-stats-auth\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.094929 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.094594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.094929 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.094822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-cert\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:22.099329 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.099305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvv7\" (UniqueName: \"kubernetes.io/projected/8dfd0ee8-e42a-4bbd-b355-44262861da53-kube-api-access-dcvv7\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:22.101631 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.101607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjd2s\" (UniqueName: \"kubernetes.io/projected/7afda5df-8528-4304-aa1d-156aefde0156-kube-api-access-hjd2s\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.105290 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.105266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9qc\" (UniqueName: \"kubernetes.io/projected/b8edf410-4aa2-42ea-bf3f-016002b74ffa-kube-api-access-8t9qc\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:22.106429 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.106404 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.106682 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.106666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:22.106741 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.106724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmq4\" (UniqueName: \"kubernetes.io/projected/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9-kube-api-access-2kmq4\") pod \"volume-data-source-validator-58c9f5f46d-q7688\" (UID: \"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:22.107008 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.106989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqq7\" (UniqueName: \"kubernetes.io/projected/aad03e4e-7139-463f-baf7-752544c6763f-kube-api-access-bvqq7\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.110560 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.110451 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgzm\" (UniqueName: \"kubernetes.io/projected/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-kube-api-access-nvgzm\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.110560 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.110505 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bjc\" (UniqueName: \"kubernetes.io/projected/3ce54792-3b37-4895-a39a-d233a7e3d4ee-kube-api-access-q4bjc\") pod \"kas-connection-checker-685895c7c9-gftjj\" (UID: \"3ce54792-3b37-4895-a39a-d233a7e3d4ee\") " pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:22.111668 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.111645 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.112662 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.112622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fsgl\" (UniqueName: \"kubernetes.io/projected/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-kube-api-access-7fsgl\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:22.112662 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.112636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.112884 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.112852 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.113399 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.113376 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlrr\" (UniqueName: \"kubernetes.io/projected/01689403-7698-4f38-b4c7-8a3603742ca8-kube-api-access-gdlrr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.113497 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.113451 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrjl\" (UniqueName: \"kubernetes.io/projected/90d658ad-d248-4853-8bd8-b64076ec9968-kube-api-access-mwrjl\") pod \"network-check-source-6f78d59f99-cbfsw\" (UID: \"90d658ad-d248-4853-8bd8-b64076ec9968\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:22.115003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.114558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:22.115003 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.114958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4lb\" (UniqueName: \"kubernetes.io/projected/67c4f288-6571-456f-8aa8-d6f5947eb14d-kube-api-access-5c4lb\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.115907 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.115888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8jn\" (UniqueName: \"kubernetes.io/projected/b06c9464-bdf9-493a-8d44-37cca42409d1-kube-api-access-6k8jn\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.125715 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.125570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:22.143760 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.143738 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:22.157201 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.156183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:22.167581 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.163491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:22.175842 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.175643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:22.210763 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.206949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:22.210763 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.207468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:22.214380 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.214352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:22.242761 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.238730 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:22.360224 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.360120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:22.418418 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.418395 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k5t2f"] Oct 31 22:49:22.420966 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.420945 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-khbjk"] Oct 31 22:49:22.429144 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.429122 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4043168_4d17_4609_bc2f_37951b7480b4.slice/crio-5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963 WatchSource:0}: Error finding container 5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963: Status 404 returned error can't find the container with id 5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963 Oct 31 22:49:22.442037 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.442012 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c14a7d4_f9a1_4e58_8ee5_8455aecf175e.slice/crio-8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b WatchSource:0}: Error finding container 8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b: Status 404 returned error can't find the container with id 8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b Oct 31 22:49:22.487524 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.487492 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gftjj"] Oct 31 22:49:22.491075 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.491038 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9"] Oct 31 22:49:22.492230 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.492169 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce54792_3b37_4895_a39a_d233a7e3d4ee.slice/crio-e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f WatchSource:0}: Error finding container e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f: Status 404 returned error can't find the container with id e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f Oct 31 22:49:22.500626 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.499827 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01689403_7698_4f38_b4c7_8a3603742ca8.slice/crio-3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6 WatchSource:0}: Error finding container 3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6: Status 404 returned error can't find the container with id 3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6 Oct 31 22:49:22.519029 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.518967 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fj2lt"] Oct 31 22:49:22.531390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.531331 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688"] Oct 31 22:49:22.539485 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.539458 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1c440e_90c0_4411_84a4_a5bfaf91d3e9.slice/crio-4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f WatchSource:0}: Error finding container 4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f: Status 404 returned error can't find the container with id 4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f Oct 31 22:49:22.602497 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.602469 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw"] Oct 31 22:49:22.605364 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.605330 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d658ad_d248_4853_8bd8_b64076ec9968.slice/crio-8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d WatchSource:0}: Error finding container 8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d: Status 404 returned error can't find the container with id 8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d Oct 31 22:49:22.606751 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.606687 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-vkldg"] Oct 31 22:49:22.611946 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.610487 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l"] Oct 31 22:49:22.611946 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.611237 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2hnp"] Oct 31 22:49:22.611946 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.611572 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ea1ea8_83a9_4c99_a124_5bf66d1f26b5.slice/crio-00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48 WatchSource:0}: Error finding container 00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48: Status 404 returned error can't find the container with id 00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48 Oct 31 22:49:22.612623 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:22.612727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:22.612727 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:22.612727 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612702 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 22:49:22.612870 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612761 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert podName:eec52f11-3785-48d2-92a1-cbd30a6047e2 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.612741546 +0000 UTC m=+32.196095243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qv9l7" (UID: "eec52f11-3785-48d2-92a1-cbd30a6047e2") : secret "networking-console-plugin-cert" not found Oct 31 22:49:22.612870 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:22.612870 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612785 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:22.612870 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612799 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:22.612870 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.612870 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612841 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.612826378 +0000 UTC m=+32.196180072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612909 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.612899039 +0000 UTC m=+32.196252746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.612990 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.612998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613028 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls podName:67c4f288-6571-456f-8aa8-d6f5947eb14d nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.613016375 +0000 UTC m=+32.196370074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls") pod "dns-default-xctjg" (UID: "67c4f288-6571-456f-8aa8-d6f5947eb14d") : secret "dns-default-metrics-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613090 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613102 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613135 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.613123895 +0000 UTC m=+32.196477593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613092 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613146 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.613161475 +0000 UTC m=+32.196515168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : secret "router-metrics-certs-default" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.613175971 +0000 UTC m=+32.196529666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613232 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 22:49:22.613280 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:22.613264 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls podName:b8edf410-4aa2-42ea-bf3f-016002b74ffa nodeName:}" failed. No retries permitted until 2026-10-31 22:49:23.613254561 +0000 UTC m=+32.196608259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5blk9" (UID: "b8edf410-4aa2-42ea-bf3f-016002b74ffa") : secret "samples-operator-tls" not found Oct 31 22:49:22.617155 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.617129 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06c9464_bdf9_493a_8d44_37cca42409d1.slice/crio-d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d WatchSource:0}: Error finding container d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d: Status 404 returned error can't find the container with id d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d Oct 31 22:49:22.617991 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:22.617971 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9191bfb_6d65_4d74_8948_fcdb9ad57b76.slice/crio-482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae WatchSource:0}: Error finding container 482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae: Status 404 returned error can't find the container with id 482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae Oct 31 22:49:22.915136 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.915056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:22.915136 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.915094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:22.915336 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.915153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:22.918595 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.918568 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 22:49:22.918595 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.918592 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 22:49:22.918773 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.918570 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\"" Oct 31 22:49:22.918773 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.918611 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\"" Oct 31 22:49:22.918773 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:22.918686 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\"" Oct 31 22:49:23.137243 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.137207 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" event={"ID":"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5","Type":"ContainerStarted","Data":"00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48"} Oct 31 22:49:23.138260 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.138238 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" event={"ID":"aad03e4e-7139-463f-baf7-752544c6763f","Type":"ContainerStarted","Data":"88a83900951ee18ae1b9dbd0ae556bcda488684b10acbba950bf6bfeced490cf"} Oct 31 22:49:23.139711 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.139686 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" event={"ID":"e4043168-4d17-4609-bc2f-37951b7480b4","Type":"ContainerStarted","Data":"5976a89397958bbc58e49177b6ead95dd6004f5ec610d0239d19447b611ada49"} Oct 31 22:49:23.139808 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.139726 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" event={"ID":"e4043168-4d17-4609-bc2f-37951b7480b4","Type":"ContainerStarted","Data":"5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963"} Oct 31 22:49:23.140839 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.140816 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" event={"ID":"90d658ad-d248-4853-8bd8-b64076ec9968","Type":"ContainerStarted","Data":"8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d"} Oct 31 22:49:23.141890 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.141863 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2hnp" event={"ID":"f9191bfb-6d65-4d74-8948-fcdb9ad57b76","Type":"ContainerStarted","Data":"482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae"} Oct 31 22:49:23.142815 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.142798 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" event={"ID":"b06c9464-bdf9-493a-8d44-37cca42409d1","Type":"ContainerStarted","Data":"d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d"} Oct 31 22:49:23.143850 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.143833 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" event={"ID":"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9","Type":"ContainerStarted","Data":"4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f"} Oct 31 22:49:23.144814 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.144792 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" event={"ID":"01689403-7698-4f38-b4c7-8a3603742ca8","Type":"ContainerStarted","Data":"3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6"} Oct 31 22:49:23.146170 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.146145 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-khbjk" event={"ID":"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e","Type":"ContainerStarted","Data":"12544b9ff7c6b9657b0cd0a6aa94f49208a90e42a0f74835e4cd96c91d4d2937"} Oct 31 22:49:23.146255 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.146180 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-khbjk" event={"ID":"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e","Type":"ContainerStarted","Data":"8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b"} Oct 31 22:49:23.147493 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.147473 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gftjj" event={"ID":"3ce54792-3b37-4895-a39a-d233a7e3d4ee","Type":"ContainerStarted","Data":"87393ec79bc5c807337665a346010ee58dda69c26da349943aaef4eafea11044"} Oct 31 22:49:23.147576 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.147510 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gftjj" event={"ID":"3ce54792-3b37-4895-a39a-d233a7e3d4ee","Type":"ContainerStarted","Data":"e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f"} Oct 31 22:49:23.155978 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.155936 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" podStartSLOduration=383.155922222 podStartE2EDuration="6m23.155922222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:23.155207212 +0000 UTC m=+31.738560922" watchObservedRunningTime="2026-10-31 22:49:23.155922222 +0000 UTC m=+31.739275939" Oct 31 22:49:23.175732 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.170986 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-khbjk" podStartSLOduration=383.170972345 podStartE2EDuration="6m23.170972345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:23.170188825 +0000 UTC m=+31.753542543" watchObservedRunningTime="2026-10-31 22:49:23.170972345 +0000 UTC m=+31.754326061" Oct 31 22:49:23.191686 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.191647 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gftjj" podStartSLOduration=383.191637239 podStartE2EDuration="6m23.191637239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:23.190466072 +0000 UTC m=+31.773819790" watchObservedRunningTime="2026-10-31 22:49:23.191637239 +0000 UTC m=+31.774990970" Oct 31 22:49:23.349493 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.349460 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:23.349666 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.349643 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:49:23.350878 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.350850 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:23.490280 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.490184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:23.623521 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:23.623685 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:23.623685 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:23.623685 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623632 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:23.623685 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:23.623886 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:23.623886 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:23.623886 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:23.623827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:23.624018 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.623951 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 22:49:23.624018 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert podName:eec52f11-3785-48d2-92a1-cbd30a6047e2 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.623995953 +0000 UTC m=+34.207349669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qv9l7" (UID: "eec52f11-3785-48d2-92a1-cbd30a6047e2") : secret "networking-console-plugin-cert" not found Oct 31 22:49:23.624446 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624424 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 22:49:23.624524 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624481 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls podName:67c4f288-6571-456f-8aa8-d6f5947eb14d nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.624465226 +0000 UTC m=+34.207818936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls") pod "dns-default-xctjg" (UID: "67c4f288-6571-456f-8aa8-d6f5947eb14d") : secret "dns-default-metrics-tls" not found Oct 31 22:49:23.624607 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624537 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:23.624607 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.624574693 +0000 UTC m=+34.207928395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:23.624700 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624650 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:23.624700 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624661 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:23.624700 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624695 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.6246834 +0000 UTC m=+34.208037111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:23.624806 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624758 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.624746521 +0000 UTC m=+34.208100229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:23.624866 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624811 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:23.624866 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624821 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:23.624866 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624847 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.624837821 +0000 UTC m=+34.208191531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:23.624966 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624897 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 22:49:23.624966 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624928 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls podName:b8edf410-4aa2-42ea-bf3f-016002b74ffa nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.624917718 +0000 UTC m=+34.208271427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5blk9" (UID: "b8edf410-4aa2-42ea-bf3f-016002b74ffa") : secret "samples-operator-tls" not found Oct 31 22:49:23.625021 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.624974 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 22:49:23.625021 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:23.625003 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:25.624992292 +0000 UTC m=+34.208346003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : secret "router-metrics-certs-default" not found Oct 31 22:49:25.161710 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.161596 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" event={"ID":"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9","Type":"ContainerStarted","Data":"8deb5d0efff6c2318ceab42a4201c95fbb982d0d6882d7221190f4387400df79"} Oct 31 22:49:25.329269 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.329181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:25.464881 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.464767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:25.645888 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.645749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:25.645888 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.645817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:25.645888 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.645867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:25.646151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.645897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:25.646151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.645928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:25.646151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.645989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:25.646151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.646072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:25.646151 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.646135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:25.646400 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.646170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:25.646400 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.646314 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:25.646400 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.646382 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.646363206 +0000 UTC m=+38.229716925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:25.646886 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.646860 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:25.646886 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.646881 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:25.647034 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.646920 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.646908658 +0000 UTC m=+38.230262351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:25.647079 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647040 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 22:49:25.647079 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647077 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.647066537 +0000 UTC m=+38.230420231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : secret "router-metrics-certs-default" not found Oct 31 22:49:25.647152 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647112 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:25.647152 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647119 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:25.647152 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647126 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 22:49:25.647152 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647143 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.647133968 +0000 UTC m=+38.230487662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:25.647341 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647181 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 22:49:25.647341 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls podName:67c4f288-6571-456f-8aa8-d6f5947eb14d nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.647169479 +0000 UTC m=+38.230523187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls") pod "dns-default-xctjg" (UID: "67c4f288-6571-456f-8aa8-d6f5947eb14d") : secret "dns-default-metrics-tls" not found Oct 31 22:49:25.647341 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647208 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls podName:b8edf410-4aa2-42ea-bf3f-016002b74ffa nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.647199823 +0000 UTC m=+38.230553517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5blk9" (UID: "b8edf410-4aa2-42ea-bf3f-016002b74ffa") : secret "samples-operator-tls" not found Oct 31 22:49:25.647341 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647257 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 22:49:25.647341 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647294 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert podName:eec52f11-3785-48d2-92a1-cbd30a6047e2 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.647282181 +0000 UTC m=+38.230635892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qv9l7" (UID: "eec52f11-3785-48d2-92a1-cbd30a6047e2") : secret "networking-console-plugin-cert" not found Oct 31 22:49:25.647698 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.647683 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.647664493 +0000 UTC m=+38.231018210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:25.650451 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.650427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:25.671390 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.671365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:25.747267 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.747241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:25.747393 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.747362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:25.747891 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.747871 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 22:49:25.747977 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:25.747932 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:57.747918506 +0000 UTC m=+66.331272200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : secret "metrics-daemon-secret" not found Oct 31 22:49:25.750431 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.750383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:25.987915 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:25.987641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:26.011286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:26.011190 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" podStartSLOduration=356.708403747 podStartE2EDuration="5m59.011171558s" totalImagesPullingTime="2.302767811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:25.179022186 +0000 UTC m=+33.762375902" watchObservedRunningTime="2026-10-31 22:49:26.011171558 +0000 UTC m=+34.594525275" Oct 31 22:49:26.012273 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:26.012241 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xrfs4"] Oct 31 22:49:26.021643 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:26.021607 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8955fd_a862_461a_bce9_a5be688358ef.slice/crio-4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1 WatchSource:0}: Error finding container 4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1: Status 404 returned error can't find the container with id 4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1 Oct 31 22:49:26.183169 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:26.183129 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xrfs4" event={"ID":"ce8955fd-a862-461a-bce9-a5be688358ef","Type":"ContainerStarted","Data":"4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1"} Oct 31 22:49:26.340375 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:26.339247 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56mmn"] Oct 31 22:49:26.355477 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:26.355434 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7533bb1d_7252_4b45_b915_969a3f216ba4.slice/crio-fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d WatchSource:0}: Error finding container fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d: Status 404 returned error can't find the container with id fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d Oct 31 22:49:26.863459 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:26.863424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:27.187906 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:27.187816 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56mmn" event={"ID":"7533bb1d-7252-4b45-b915-969a3f216ba4","Type":"ContainerStarted","Data":"fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d"} Oct 31 22:49:28.194426 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.194296 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" event={"ID":"01689403-7698-4f38-b4c7-8a3603742ca8","Type":"ContainerStarted","Data":"0d163879de497c028af2f928ae7f07a01c467c75afa8cca23ce814aca7998f91"} Oct 31 22:49:28.196714 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.196676 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/0.log" Oct 31 22:49:28.196845 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.196716 2576 generic.go:356] "Generic (PLEG): container finished" podID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerID="702ebf88ebc55aa34bb080b4924b92e18e8522f35ab1879d8a63a659fc875d77" exitCode=255 Oct 31 22:49:28.196845 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.196782 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" event={"ID":"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5","Type":"ContainerDied","Data":"702ebf88ebc55aa34bb080b4924b92e18e8522f35ab1879d8a63a659fc875d77"} Oct 31 22:49:28.197262 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.197046 2576 scope.go:122] "RemoveContainer" containerID="702ebf88ebc55aa34bb080b4924b92e18e8522f35ab1879d8a63a659fc875d77" Oct 31 22:49:28.202286 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.202260 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" event={"ID":"aad03e4e-7139-463f-baf7-752544c6763f","Type":"ContainerStarted","Data":"7033cf2ae1df5b527a28b7b3121dd48d89edc4268291194b2749359f86e03372"} Oct 31 22:49:28.205036 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.205006 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2hnp" event={"ID":"f9191bfb-6d65-4d74-8948-fcdb9ad57b76","Type":"ContainerStarted","Data":"de650f82672b372ab642d05773cf4c61df7a6254450ea744cbc84811260dd195"} Oct 31 22:49:28.208240 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.208215 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" event={"ID":"b06c9464-bdf9-493a-8d44-37cca42409d1","Type":"ContainerStarted","Data":"f91d48dabb17b85c59b1c4a65404adf3c4841a9ff4f4ae68854f0739e7a009c4"} Oct 31 22:49:28.212881 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.212843 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" podStartSLOduration=356.19860595 podStartE2EDuration="6m1.212832939s" totalImagesPullingTime="5.014226989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:28.211949655 +0000 UTC m=+36.795303372" watchObservedRunningTime="2026-10-31 22:49:28.212832939 +0000 UTC m=+36.796186656" Oct 31 22:49:28.230919 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.230785 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" podStartSLOduration=356.238343487 podStartE2EDuration="6m1.230767017s" totalImagesPullingTime="4.99242353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:28.229531665 +0000 UTC m=+36.812885381" watchObservedRunningTime="2026-10-31 22:49:28.230767017 +0000 UTC m=+36.814120734" Oct 31 22:49:28.252678 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.252597 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r2hnp" podStartSLOduration=2.056061955 podStartE2EDuration="7.252580386s" totalImagesPullingTime="5.196518431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:28.251075338 +0000 UTC m=+36.834429056" watchObservedRunningTime="2026-10-31 22:49:28.252580386 +0000 UTC m=+36.835934104" Oct 31 22:49:28.270109 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.269518 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" podStartSLOduration=356.348228897 podStartE2EDuration="6m1.269500988s" totalImagesPullingTime="4.921272091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:28.267411702 +0000 UTC m=+36.850765421" watchObservedRunningTime="2026-10-31 22:49:28.269500988 +0000 UTC m=+36.852854704" Oct 31 22:49:28.496306 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.496235 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wtb2p"] Oct 31 22:49:28.513678 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.513649 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wtb2p"] Oct 31 22:49:28.513947 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.513789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.518752 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.518730 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4xsrk-mwv8t"] Oct 31 22:49:28.533725 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.533678 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4xsrk-mwv8t"] Oct 31 22:49:28.533853 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.533796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.539264 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.539240 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 22:49:28.539432 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.539372 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9xrd\"" Oct 31 22:49:28.561319 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.561287 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 22:49:28.576929 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.576486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:28.579771 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.579742 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-447ws\"" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.579954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/756d759c-3587-4ff9-9abb-f24c46f64077-crio-socket\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntq5r\" (UniqueName: \"kubernetes.io/projected/756d759c-3587-4ff9-9abb-f24c46f64077-kube-api-access-ntq5r\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/756d759c-3587-4ff9-9abb-f24c46f64077-data-volume\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.580381 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.580236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.681427 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.681626 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.681626 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/756d759c-3587-4ff9-9abb-f24c46f64077-crio-socket\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.681626 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntq5r\" (UniqueName: \"kubernetes.io/projected/756d759c-3587-4ff9-9abb-f24c46f64077-kube-api-access-ntq5r\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.681626 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/756d759c-3587-4ff9-9abb-f24c46f64077-data-volume\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.681812 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.681812 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.681812 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.681812 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.681785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.681993 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:28.681922 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:28.681993 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:28.681982 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.18196391 +0000 UTC m=+37.765317625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:28.683181 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.682409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/756d759c-3587-4ff9-9abb-f24c46f64077-data-volume\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.683181 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.682895 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.683181 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.682945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.683181 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.682970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/756d759c-3587-4ff9-9abb-f24c46f64077-crio-socket\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.683181 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:28.683022 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 22:49:28.683181 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:28.683089 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:29.183070256 +0000 UTC m=+37.766423955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : configmap "kube-rbac-proxy" not found Oct 31 22:49:28.684442 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.684416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.707612 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.707539 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.709306 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.709260 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntq5r\" (UniqueName: \"kubernetes.io/projected/756d759c-3587-4ff9-9abb-f24c46f64077-kube-api-access-ntq5r\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:28.739363 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.739332 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm"] Oct 31 22:49:28.740466 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.740441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:28.767454 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.767390 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f"] Oct 31 22:49:28.769292 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.767683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:28.771825 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.771789 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:28.771921 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.771890 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 22:49:28.772275 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.772259 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jfvz2\"" Oct 31 22:49:28.781453 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.781432 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm"] Oct 31 22:49:28.781588 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.781465 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f"] Oct 31 22:49:28.781664 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.781594 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:28.883666 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.883636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczl4\" (UniqueName: \"kubernetes.io/projected/c8f7392b-faf8-43ad-a96b-7e95a95d7792-kube-api-access-dczl4\") pod \"migrator-66df5d496b-ttq5f\" (UID: \"c8f7392b-faf8-43ad-a96b-7e95a95d7792\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:28.883815 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.883721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5c8\" (UniqueName: \"kubernetes.io/projected/39a8980d-f812-4330-84e7-c406c8afa702-kube-api-access-kc5c8\") pod \"migrator-66df5d496b-tllzm\" (UID: \"39a8980d-f812-4330-84e7-c406c8afa702\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:28.889100 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.889074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:28.890946 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.890929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:28.985029 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.984995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczl4\" (UniqueName: \"kubernetes.io/projected/c8f7392b-faf8-43ad-a96b-7e95a95d7792-kube-api-access-dczl4\") pod \"migrator-66df5d496b-ttq5f\" (UID: \"c8f7392b-faf8-43ad-a96b-7e95a95d7792\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:28.985165 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.985076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5c8\" (UniqueName: \"kubernetes.io/projected/39a8980d-f812-4330-84e7-c406c8afa702-kube-api-access-kc5c8\") pod \"migrator-66df5d496b-tllzm\" (UID: \"39a8980d-f812-4330-84e7-c406c8afa702\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:28.997043 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.996989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczl4\" (UniqueName: \"kubernetes.io/projected/c8f7392b-faf8-43ad-a96b-7e95a95d7792-kube-api-access-dczl4\") pod \"migrator-66df5d496b-ttq5f\" (UID: \"c8f7392b-faf8-43ad-a96b-7e95a95d7792\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:28.997171 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:28.997043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5c8\" (UniqueName: \"kubernetes.io/projected/39a8980d-f812-4330-84e7-c406c8afa702-kube-api-access-kc5c8\") pod \"migrator-66df5d496b-tllzm\" (UID: \"39a8980d-f812-4330-84e7-c406c8afa702\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:29.028448 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.028329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:29.092564 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.092519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:29.116359 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.116330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:29.189984 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.187967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:29.189984 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.188054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:29.189984 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.188230 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 22:49:29.189984 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.188290 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:30.188272347 +0000 UTC m=+38.771626046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : configmap "kube-rbac-proxy" not found Oct 31 22:49:29.189984 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.188723 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:29.189984 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.188778 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:30.188761431 +0000 UTC m=+38.772115142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:29.216036 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.216009 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:49:29.216525 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.216503 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/0.log" Oct 31 22:49:29.216623 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.216561 2576 generic.go:356] "Generic (PLEG): container finished" podID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" exitCode=255 Oct 31 22:49:29.216772 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.216748 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" event={"ID":"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5","Type":"ContainerDied","Data":"b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5"} Oct 31 22:49:29.216844 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.216813 2576 scope.go:122] "RemoveContainer" containerID="702ebf88ebc55aa34bb080b4924b92e18e8522f35ab1879d8a63a659fc875d77" Oct 31 22:49:29.218719 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.217699 2576 scope.go:122] "RemoveContainer" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" Oct 31 22:49:29.218719 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.217930 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vkldg_openshift-console-operator(12ea1ea8-83a9-4c99-a124-5bf66d1f26b5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" Oct 31 22:49:29.303410 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.302289 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8g88j"] Oct 31 22:49:29.320324 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.320298 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8g88j"] Oct 31 22:49:29.320456 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.320396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.324278 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.324250 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 22:49:29.324428 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.324338 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 22:49:29.324490 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.324476 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 22:49:29.324676 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.324660 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-stxgn\"" Oct 31 22:49:29.324739 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.324728 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 22:49:29.390979 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.390948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-cabundle\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.391122 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.390988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjq8\" (UniqueName: \"kubernetes.io/projected/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-kube-api-access-nxjq8\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.391122 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.391109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-key\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.410354 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.410328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:29.425026 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.425000 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4xsrk-mwv8t"] Oct 31 22:49:29.428613 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:29.428585 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a36a016_71e1_4a23_b4ca_768e186091e3.slice/crio-a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417 WatchSource:0}: Error finding container a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417: Status 404 returned error can't find the container with id a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417 Oct 31 22:49:29.454062 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.454001 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm"] Oct 31 22:49:29.456742 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.456719 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f"] Oct 31 22:49:29.466148 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:29.466113 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a8980d_f812_4330_84e7_c406c8afa702.slice/crio-84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf WatchSource:0}: Error finding container 84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf: Status 404 returned error can't find the container with id 84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf Oct 31 22:49:29.466745 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:29.466636 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f7392b_faf8_43ad_a96b_7e95a95d7792.slice/crio-25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3 WatchSource:0}: Error finding container 25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3: Status 404 returned error can't find the container with id 25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3 Oct 31 22:49:29.492029 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.492010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-key\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.492524 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.492224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-cabundle\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.492524 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.492258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjq8\" (UniqueName: \"kubernetes.io/projected/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-kube-api-access-nxjq8\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.493037 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.493016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-cabundle\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.494914 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.494871 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-key\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.504335 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.504309 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjq8\" (UniqueName: \"kubernetes.io/projected/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-kube-api-access-nxjq8\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.549173 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.548890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:29.648807 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.648771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.706944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.707004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.707036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.707062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707066 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707140 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert podName:eec52f11-3785-48d2-92a1-cbd30a6047e2 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.707120028 +0000 UTC m=+46.290473726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qv9l7" (UID: "eec52f11-3785-48d2-92a1-cbd30a6047e2") : secret "networking-console-plugin-cert" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.707167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707195 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707207 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.707221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707770 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707791 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707242 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.707231206 +0000 UTC m=+46.290584934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.707960 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708032 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.708009753 +0000 UTC m=+46.291363453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:29.717572 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708066 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.708049871 +0000 UTC m=+46.291403568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708084 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls podName:67c4f288-6571-456f-8aa8-d6f5947eb14d nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.708074751 +0000 UTC m=+46.291428471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls") pod "dns-default-xctjg" (UID: "67c4f288-6571-456f-8aa8-d6f5947eb14d") : secret "dns-default-metrics-tls" not found Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.708121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.708195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708330 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708373 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.708362301 +0000 UTC m=+46.291716013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : secret "router-metrics-certs-default" not found Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708450 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls podName:b8edf410-4aa2-42ea-bf3f-016002b74ffa nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.708473987 +0000 UTC m=+46.291827720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5blk9" (UID: "b8edf410-4aa2-42ea-bf3f-016002b74ffa") : secret "samples-operator-tls" not found Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708580 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:29.718468 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:29.708622 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:37.70860388 +0000 UTC m=+46.291957577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:29.974853 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:29.974730 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8g88j"] Oct 31 22:49:29.979567 ip-10-0-133-222 kubenswrapper[2576]: W1031 22:49:29.979498 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d95fc77_4bb6_4464_97f2_2bd5500d88e1.slice/crio-282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842 WatchSource:0}: Error finding container 282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842: Status 404 returned error can't find the container with id 282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842 Oct 31 22:49:30.212555 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.212519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:30.212717 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:30.212635 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 22:49:30.212717 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:30.212712 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:32.212692701 +0000 UTC m=+40.796046418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : configmap "kube-rbac-proxy" not found Oct 31 22:49:30.212821 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.212763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:30.212888 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:30.212876 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:30.212923 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:30.212910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:32.212899214 +0000 UTC m=+40.796252922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:30.222978 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.222958 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:49:30.223353 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.223313 2576 scope.go:122] "RemoveContainer" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" Oct 31 22:49:30.223519 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:30.223494 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vkldg_openshift-console-operator(12ea1ea8-83a9-4c99-a124-5bf66d1f26b5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" Oct 31 22:49:30.225169 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.225108 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" event={"ID":"90d658ad-d248-4853-8bd8-b64076ec9968","Type":"ContainerStarted","Data":"f08018c6a2218cd62f198e805ed7c2b07a8eafe52b5e48b5911cbf4616e7c5f8"} Oct 31 22:49:30.227063 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.227041 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" event={"ID":"1d95fc77-4bb6-4464-97f2-2bd5500d88e1","Type":"ContainerStarted","Data":"38b786aacf5f0cceaebf7b9d9782c5811ec3d8c7d12414dff25f86d9759a6f49"} Oct 31 22:49:30.227164 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.227084 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" event={"ID":"1d95fc77-4bb6-4464-97f2-2bd5500d88e1","Type":"ContainerStarted","Data":"282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842"} Oct 31 22:49:30.228273 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.228243 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" event={"ID":"c8f7392b-faf8-43ad-a96b-7e95a95d7792","Type":"ContainerStarted","Data":"25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3"} Oct 31 22:49:30.229450 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.229426 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" event={"ID":"39a8980d-f812-4330-84e7-c406c8afa702","Type":"ContainerStarted","Data":"84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf"} Oct 31 22:49:30.231188 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.231155 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" event={"ID":"9a36a016-71e1-4a23-b4ca-768e186091e3","Type":"ContainerStarted","Data":"62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416"} Oct 31 22:49:30.231265 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.231196 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" event={"ID":"9a36a016-71e1-4a23-b4ca-768e186091e3","Type":"ContainerStarted","Data":"a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417"} Oct 31 22:49:30.259317 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.258779 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" podStartSLOduration=1.2587651100000001 podStartE2EDuration="1.25876511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:30.258239052 +0000 UTC m=+38.841592769" watchObservedRunningTime="2026-10-31 22:49:30.25876511 +0000 UTC m=+38.842118828" Oct 31 22:49:30.274461 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.274399 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" podStartSLOduration=349.731730754 podStartE2EDuration="5m56.274382027s" totalImagesPullingTime="6.542651273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:30.274222864 +0000 UTC m=+38.857576581" watchObservedRunningTime="2026-10-31 22:49:30.274382027 +0000 UTC m=+38.857735744" Oct 31 22:49:30.292044 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.292002 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" podStartSLOduration=2.291989962 podStartE2EDuration="2.291989962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:30.290637285 +0000 UTC m=+38.873991003" watchObservedRunningTime="2026-10-31 22:49:30.291989962 +0000 UTC m=+38.875343677" Oct 31 22:49:30.427991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.427962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:30.624347 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.624237 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:30.771301 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.771228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:30.909866 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:30.909724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:31.081063 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:31.081028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:31.249284 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:31.249191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:31.251183 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:31.251153 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56mmn" event={"ID":"7533bb1d-7252-4b45-b915-969a3f216ba4","Type":"ContainerStarted","Data":"ce7f81f135773c66c4f47d5d84368b431674522c4698910e55a3b27f74678026"} Oct 31 22:49:31.251991 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:31.251969 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:31.274174 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:31.274124 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-56mmn" podStartSLOduration=35.301456718 podStartE2EDuration="39.274107511s" totalImagesPullingTime="3.972650793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:31.272317243 +0000 UTC m=+39.855670959" watchObservedRunningTime="2026-10-31 22:49:31.274107511 +0000 UTC m=+39.857461229" Oct 31 22:49:32.208130 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.208037 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:32.208283 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.208139 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:32.208604 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.208585 2576 scope.go:122] "RemoveContainer" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" Oct 31 22:49:32.208825 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:32.208802 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-vkldg_openshift-console-operator(12ea1ea8-83a9-4c99-a124-5bf66d1f26b5)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" Oct 31 22:49:32.236138 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.236105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:32.236301 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.236260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:32.236387 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:32.236373 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:32.236451 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:32.236433 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.23641811 +0000 UTC m=+44.819771805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:32.236794 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:32.236771 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 22:49:32.236911 ip-10-0-133-222 kubenswrapper[2576]: E1031 22:49:32.236824 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.236809383 +0000 UTC m=+44.820163094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : configmap "kube-rbac-proxy" not found Oct 31 22:49:32.248419 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.248391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:32.255666 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.255634 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" event={"ID":"c8f7392b-faf8-43ad-a96b-7e95a95d7792","Type":"ContainerStarted","Data":"3c5fec540a583feccf3b2f7e411365124dc0f1577e9e0a2795f3cbf409ceaac6"} Oct 31 22:49:32.257166 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.257140 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" event={"ID":"39a8980d-f812-4330-84e7-c406c8afa702","Type":"ContainerStarted","Data":"71bad2b7241c21eced4e24c559d7e5cc745746d0d19a05e20eb9b7d013eaad7e"} Oct 31 22:49:32.278205 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.278159 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" podStartSLOduration=1.883178733 podStartE2EDuration="4.278147425s" totalImagesPullingTime="2.394968692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:49:32.276946727 +0000 UTC m=+40.860300457" watchObservedRunningTime="2026-10-31 22:49:32.278147425 +0000 UTC m=+40.861501141" Oct 31 22:49:32.443973 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.443933 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:32.647321 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.647241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:32.897833 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:32.897757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:33.056334 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:33.056299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:33.208694 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:33.208628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:33.694627 ip-10-0-133-222 kubenswrapper[2576]: I1031 22:49:33.694539 2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:49:33.694724 ip-10-0-133-222 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 22:49:33.718925 ip-10-0-133-222 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 22:49:33.719096 ip-10-0-133-222 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 22:49:33.719145 ip-10-0-133-222 systemd[1]: kubelet.service: Consumed 3.096s CPU time. Oct 31 22:49:33.729199 ip-10-0-133-222 systemd[1]: Starting Kubernetes Kubelet... Oct 31 22:49:33.844051 ip-10-0-133-222 kubenswrapper[5043]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:49:33.844051 ip-10-0-133-222 kubenswrapper[5043]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 22:49:33.844051 ip-10-0-133-222 kubenswrapper[5043]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:49:33.844051 ip-10-0-133-222 kubenswrapper[5043]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 22:49:33.846036 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.845956 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.846036 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846036 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846042 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846047 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846050 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846054 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846057 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846060 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846063 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846066 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846069 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846071 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846074 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846076 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846079 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846082 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846084 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846087 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846090 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846092 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.846120 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846095 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846101 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846104 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846106 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846109 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846112 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846114 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846117 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846119 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846123 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846126 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846129 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846132 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846134 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846137 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846140 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846143 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846147 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846151 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.846701 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846154 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846157 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846160 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846162 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846165 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846168 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846170 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846174 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846176 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846179 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846182 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846184 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846187 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846190 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846192 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846195 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846198 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846200 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846203 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846206 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.847151 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846208 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846213 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846217 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846221 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846225 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846229 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846233 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846238 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846242 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846245 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846247 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846250 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846253 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846256 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846259 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846262 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846264 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846267 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846270 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846272 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.847645 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846275 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846277 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846280 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846282 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846285 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846288 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846291 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846293 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846296 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846298 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846301 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846304 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846307 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846312 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846315 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846317 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846320 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846323 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846325 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846328 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.848143 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846330 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846333 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846336 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846338 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846341 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846343 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.846346 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847129 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847136 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847139 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847142 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847146 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847148 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847151 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847154 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847157 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847160 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847163 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847166 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847169 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.848676 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847171 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847176 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847179 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847182 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847185 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847188 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847191 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847193 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847196 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847199 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847202 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847204 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847207 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847209 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847212 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847215 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847217 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847220 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847222 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.849159 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847226 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847229 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847232 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847234 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847237 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847240 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847244 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847246 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847249 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847251 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847254 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847256 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847259 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847262 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847264 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847267 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847270 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847272 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847275 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847277 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.849678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847280 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847284 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847289 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847292 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847296 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847299 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847302 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847304 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847307 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847309 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847312 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847315 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847317 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847320 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847323 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847326 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847329 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847331 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847334 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847337 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.850156 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847340 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847342 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847345 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847348 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847351 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847353 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847356 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847358 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847361 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847364 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847366 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847369 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847372 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847374 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847377 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847379 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847382 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847384 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847387 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.850678 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847389 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847392 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847395 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847397 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847400 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847403 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847405 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847408 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847410 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847413 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847416 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847418 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847421 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847425 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847427 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847947 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847953 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847956 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847958 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847961 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.851178 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847964 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847966 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847969 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847972 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847974 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847977 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847980 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847983 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847986 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847988 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847991 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847994 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847996 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.847999 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848001 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848004 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848007 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848009 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848012 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.851672 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848020 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848023 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848026 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848028 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848036 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848039 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848042 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848044 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848049 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848051 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848054 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848056 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848059 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848062 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848064 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848067 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848069 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848072 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848075 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.852172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848077 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848080 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848082 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848085 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848088 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848090 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848093 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848095 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848098 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848101 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848104 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848106 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848109 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848111 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848115 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848117 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848120 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848124 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848126 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848129 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.852636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848131 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848135 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848138 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848141 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848143 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848146 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848149 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848153 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848155 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848158 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848162 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848166 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848169 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848172 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848176 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848180 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848184 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848187 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848190 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.853130 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848193 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848195 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848198 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848201 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848203 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848206 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848208 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848211 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848214 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848217 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848220 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848223 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848225 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848229 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848233 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848236 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848238 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848241 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848243 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848246 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.853586 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848248 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848251 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848254 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.848256 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848349 5043 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848356 5043 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848367 5043 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848372 5043 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848377 5043 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848380 5043 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848386 5043 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848390 5043 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848394 5043 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848397 5043 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848400 5043 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848403 5043 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848407 5043 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848409 5043 flags.go:64] FLAG: --cgroup-root="" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848412 5043 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848415 5043 flags.go:64] FLAG: --client-ca-file="" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848419 5043 flags.go:64] FLAG: --cloud-provider="external" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848422 5043 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848426 5043 flags.go:64] FLAG: --cluster-domain="" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848429 5043 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 22:49:33.854322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848433 5043 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848436 5043 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848440 5043 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848445 5043 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848448 5043 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848451 5043 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848454 5043 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848457 5043 flags.go:64] FLAG: --contention-profiling="false" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848461 5043 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848463 5043 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848467 5043 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848470 5043 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848474 5043 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848477 5043 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848480 5043 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848483 5043 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848486 5043 flags.go:64] FLAG: --enable-server="true" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848491 5043 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848495 5043 flags.go:64] FLAG: --event-burst="100" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848498 5043 flags.go:64] FLAG: --event-qps="50" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848501 5043 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848505 5043 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848508 5043 flags.go:64] FLAG: --eviction-hard="" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848512 5043 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 22:49:33.855099 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848515 5043 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848518 5043 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848521 5043 flags.go:64] FLAG: --eviction-soft="" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848524 5043 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848527 5043 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848530 5043 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848538 5043 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848554 5043 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848558 5043 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848561 5043 flags.go:64] FLAG: --feature-gates="" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848568 5043 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848571 5043 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848574 5043 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848577 5043 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848580 5043 flags.go:64] FLAG: --healthz-port="10248" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848584 5043 flags.go:64] FLAG: --help="false" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848586 5043 flags.go:64] FLAG: --hostname-override="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848590 5043 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848593 5043 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848596 5043 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848599 5043 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848603 5043 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848606 5043 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 22:49:33.855701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848609 5043 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848611 5043 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848614 5043 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848617 5043 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848620 5043 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848624 5043 flags.go:64] FLAG: --kube-reserved="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848627 5043 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848630 5043 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848633 5043 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848635 5043 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848639 5043 flags.go:64] FLAG: --lock-file="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848641 5043 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848644 5043 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848647 5043 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848653 5043 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848656 5043 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848659 5043 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848662 5043 flags.go:64] FLAG: --logging-format="text" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848665 5043 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848671 5043 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848674 5043 flags.go:64] FLAG: --manifest-url="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848677 5043 flags.go:64] FLAG: --manifest-url-header="" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848681 5043 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848685 5043 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848689 5043 flags.go:64] FLAG: --max-pods="110" Oct 31 22:49:33.856299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848692 5043 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848695 5043 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848698 5043 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848701 5043 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848704 5043 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848707 5043 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848710 5043 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848741 5043 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848745 5043 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848748 5043 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848751 5043 flags.go:64] FLAG: --pod-cidr="" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848755 5043 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848758 5043 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848761 5043 flags.go:64] FLAG: --pods-per-core="0" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848764 5043 flags.go:64] FLAG: --port="10250" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848767 5043 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848771 5043 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d0921356442108bb" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848775 5043 flags.go:64] FLAG: --qos-reserved="" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848778 5043 flags.go:64] FLAG: --read-only-port="10255" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848781 5043 flags.go:64] FLAG: --register-node="true" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848784 5043 flags.go:64] FLAG: --register-with-taints="" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848787 5043 flags.go:64] FLAG: --registry-burst="10" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848791 5043 flags.go:64] FLAG: --registry-qps="5" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848794 5043 flags.go:64] FLAG: --reserved-cpus="" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848797 5043 flags.go:64] FLAG: --reserved-memory="" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848801 5043 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 22:49:33.856897 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848805 5043 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848808 5043 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848811 5043 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848814 5043 flags.go:64] FLAG: --runonce="false" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848817 5043 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848820 5043 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848823 5043 flags.go:64] FLAG: --seccomp-default="false" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848826 5043 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848829 5043 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848832 5043 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848835 5043 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848838 5043 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848841 5043 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848844 5043 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848847 5043 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848850 5043 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848853 5043 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848856 5043 flags.go:64] FLAG: --system-cgroups="" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848859 5043 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848864 5043 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848867 5043 flags.go:64] FLAG: --tls-cert-file="" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848870 5043 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848875 5043 flags.go:64] FLAG: --tls-min-version="" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848878 5043 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848881 5043 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 22:49:33.857537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848884 5043 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848887 5043 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848890 5043 flags.go:64] FLAG: --v="2" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848894 5043 flags.go:64] FLAG: --version="false" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848898 5043 flags.go:64] FLAG: --vmodule="" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848902 5043 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.848906 5043 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849036 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849040 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849043 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849046 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849049 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849051 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849054 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849057 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849059 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849062 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849064 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849067 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849070 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849072 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.858192 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849075 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849078 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849080 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849083 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849085 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849088 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849091 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849093 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849097 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849100 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849102 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849105 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849108 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849111 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849113 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849116 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849118 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849121 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849127 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849130 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.858714 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849133 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849135 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849138 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849141 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849144 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849146 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849149 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849151 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849154 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849157 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849160 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849162 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849165 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849167 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849170 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849173 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849175 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849178 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849180 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849183 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.859193 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849187 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849189 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849192 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849194 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849197 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849199 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849202 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849205 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849207 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849210 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849214 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849218 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849221 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849224 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849226 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849229 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849233 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849236 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849239 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.859791 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849241 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849244 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849248 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849250 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849253 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849256 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849259 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849261 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849264 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849266 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849269 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849272 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849274 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849278 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849281 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849284 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849287 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849289 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849292 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.860328 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849294 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849297 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849300 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849303 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849305 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849308 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849311 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849313 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849316 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849319 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849321 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849324 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849326 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849329 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.849354 5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:49:33.860886 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849386 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849393 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849396 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849399 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849402 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849405 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849408 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849411 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849413 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849416 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849419 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849423 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849426 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849428 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849431 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849433 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849436 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849439 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849442 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849444 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.861566 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849447 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849449 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849452 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849455 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849458 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849460 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849463 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849466 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849468 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849471 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849473 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849476 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849478 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849483 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849485 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849488 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849491 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849493 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849496 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849498 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.862378 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849501 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849505 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849508 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849512 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849514 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849519 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849523 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849526 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849529 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849531 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849534 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849538 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849554 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849557 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849559 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849562 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849565 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849568 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849571 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.863056 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849574 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849577 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849580 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849582 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849585 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849588 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849592 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849595 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849597 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849600 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849603 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849605 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849608 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849611 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849613 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849616 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849620 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849623 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849625 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849628 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.863660 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849631 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849634 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849637 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849639 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849642 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849645 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849647 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849650 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849653 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849655 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849658 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849661 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849664 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849666 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849669 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849671 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849674 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849676 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849681 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.864153 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849684 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849687 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849689 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849692 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849695 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849697 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849700 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.849702 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.854495 5043 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.854564 5043 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.854650 5043 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.854669 5043 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855039 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855054 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855058 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855062 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855067 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855072 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855076 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.864631 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855080 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855083 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855086 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855089 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855092 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855094 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855097 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855100 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855103 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855106 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855110 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855113 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855116 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855119 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855122 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855125 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855127 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855130 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855132 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.865082 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855135 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855138 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855140 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855144 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855146 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855149 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855152 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855155 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855158 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855161 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855163 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855166 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855169 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855172 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855174 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855177 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855180 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855183 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855186 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.865535 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855188 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855191 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855194 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855197 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855199 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855202 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855205 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855207 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855210 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855212 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855215 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855217 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855220 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855223 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855226 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855228 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855231 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855234 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855237 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855240 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.866007 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855242 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855245 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855247 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855250 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855252 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855255 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855258 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855260 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855263 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855265 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855268 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855271 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855273 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855275 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855279 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855282 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855284 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855287 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855289 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855292 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.866475 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855294 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855297 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855299 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855302 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855304 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855307 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855310 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855313 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855315 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855318 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855321 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855323 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855326 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855328 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855331 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855333 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855336 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855338 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855341 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.866975 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855344 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855346 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.855376 5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855511 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855516 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855519 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855522 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855525 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855528 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855531 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855534 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855536 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855539 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855541 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855567 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855570 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.867417 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855573 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855576 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855578 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855581 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855583 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855586 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855589 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855592 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855594 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855598 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855602 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855606 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855608 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855611 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855614 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855616 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855619 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855621 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855624 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.867888 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855627 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855630 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855632 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855635 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855637 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855640 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855643 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855646 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855649 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855651 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855654 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855656 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855659 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855661 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855664 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855666 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855669 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855671 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855674 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.868366 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855676 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855679 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855681 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855684 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855687 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855690 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855692 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855695 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855697 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855700 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855703 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855705 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855708 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855711 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855714 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855716 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855719 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855721 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855724 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855731 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.868836 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855734 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855737 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855740 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855743 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855745 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855748 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855751 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855753 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855756 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855758 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855761 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855764 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855766 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855769 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855772 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855774 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855777 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855781 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855785 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.869309 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855788 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855790 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855793 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855796 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855798 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855801 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855804 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855806 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855809 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855812 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855815 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855818 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855822 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855825 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855828 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855831 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.869816 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.855857 5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855890 5043 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855894 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855897 5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855900 5043 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855903 5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855906 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855908 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855911 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855914 5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855916 5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855919 5043 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855922 5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855924 5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855927 5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855929 5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855932 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855935 5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855938 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 22:49:33.870211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855942 5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855946 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855949 5043 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855953 5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855956 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855959 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855961 5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855964 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855966 5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855969 5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855973 5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855976 5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855979 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855982 5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855985 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855987 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855990 5043 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855993 5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855996 5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.855998 5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 22:49:33.870693 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856001 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856003 5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856006 5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856009 5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856011 5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856014 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856016 5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856019 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856021 5043 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856025 5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856027 5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856031 5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856033 5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856036 5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856038 5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856041 5043 feature_gate.go:386] unrecognized feature gate: Example Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856043 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856046 5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856049 5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856051 5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 22:49:33.871344 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856055 5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856059 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856063 5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856066 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856068 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856072 5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856074 5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856077 5043 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856080 5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856082 5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856085 5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856088 5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856090 5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856093 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856095 5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856098 5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856101 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856103 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856106 5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 22:49:33.872231 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856108 5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856111 5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856115 5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856118 5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856121 5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856123 5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856126 5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856129 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856131 5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856134 5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856136 5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856139 5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856141 5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856144 5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856146 5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856150 5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856153 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856156 5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856158 5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856162 5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 22:49:33.873031 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856164 5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856167 5043 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856170 5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856172 5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856175 5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856177 5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856180 5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856183 5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:33.856186 5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.856261 5043 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857211 5043 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857260 5043 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857436 5043 server.go:1014] "Starting client certificate rotation" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857578 5043 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857632 5043 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 22:43:51 +0000 UTC" deadline="2028-07-27 17:22:35.909939944 +0000 UTC" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857671 5043 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15234h33m2.052270936s" Oct 31 22:49:33.873958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.857924 5043 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:49:33.874713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.858952 5043 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 22:49:33.874713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.860293 5043 log.go:25] "Validated CRI v1 runtime API" Oct 31 22:49:33.874713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.862389 5043 log.go:25] "Validated CRI v1 image API" Oct 31 22:49:33.874713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.863176 5043 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 22:49:33.874713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.865137 5043 fs.go:135] Filesystem UUIDs: map[5e2f145a-ebf5-4636-a5f4-1f99e29d9d6b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8102c9d4-2163-4ec4-96ca-e2899c91ab97:/dev/nvme0n1p4] Oct 31 22:49:33.874969 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.865158 5043 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88a83900951ee18ae1b9dbd0ae556bcda488684b10acbba950bf6bfeced490cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88a83900951ee18ae1b9dbd0ae556bcda488684b10acbba950bf6bfeced490cf/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01689403-7698-4f38-b4c7-8a3603742ca8/volumes/kubernetes.io~projected/kube-api-access-gdlrr:{mountpoint:/var/lib/kubelet/pods/01689403-7698-4f38-b4c7-8a3603742ca8/volumes/kubernetes.io~projected/kube-api-access-gdlrr major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01689403-7698-4f38-b4c7-8a3603742ca8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/01689403-7698-4f38-b4c7-8a3603742ca8/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~projected/kube-api-access-wzfqz:{mountpoint:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~projected/kube-api-access-wzfqz major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/volumes/kubernetes.io~projected/kube-api-access-nvgzm:{mountpoint:/var/lib/kubelet/pods/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/volumes/kubernetes.io~projected/kube-api-access-nvgzm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13e8e748-f088-43d1-8011-bf6c8f91ec59/volumes/kubernetes.io~projected/kube-api-access-nsnkd:{mountpoint:/var/lib/kubelet/pods/13e8e748-f088-43d1-8011-bf6c8f91ec59/volumes/kubernetes.io~projected/kube-api-access-nsnkd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d95fc77-4bb6-4464-97f2-2bd5500d88e1/volumes/kubernetes.io~projected/kube-api-access-nxjq8:{mountpoint:/var/lib/kubelet/pods/1d95fc77-4bb6-4464-97f2-2bd5500d88e1/volumes/kubernetes.io~projected/kube-api-access-nxjq8 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d95fc77-4bb6-4464-97f2-2bd5500d88e1/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/1d95fc77-4bb6-4464-97f2-2bd5500d88e1/volumes/kubernetes.io~secret/signing-key major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~projected/kube-api-access-plbl9:{mountpoint:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~projected/kube-api-access-plbl9 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9/volumes/kubernetes.io~projected/kube-api-access-2kmq4:{mountpoint:/var/lib/kubelet/pods/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9/volumes/kubernetes.io~projected/kube-api-access-2kmq4 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37b0f64a-6235-4472-b7e2-a2d2a08fd388/volumes/kubernetes.io~projected/kube-api-access-g9868:{mountpoint:/var/lib/kubelet/pods/37b0f64a-6235-4472-b7e2-a2d2a08fd388/volumes/kubernetes.io~projected/kube-api-access-g9868 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39a8980d-f812-4330-84e7-c406c8afa702/volumes/kubernetes.io~projected/kube-api-access-kc5c8:{mountpoint:/var/lib/kubelet/pods/39a8980d-f812-4330-84e7-c406c8afa702/volumes/kubernetes.io~projected/kube-api-access-kc5c8 major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce54792-3b37-4895-a39a-d233a7e3d4ee/volumes/kubernetes.io~projected/kube-api-access-q4bjc:{mountpoint:/var/lib/kubelet/pods/3ce54792-3b37-4895-a39a-d233a7e3d4ee/volumes/kubernetes.io~projected/kube-api-access-q4bjc major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a940d2c-fc2c-40ae-a8bd-5916089eb925/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5a940d2c-fc2c-40ae-a8bd-5916089eb925/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67c4f288-6571-456f-8aa8-d6f5947eb14d/volumes/kubernetes.io~projected/kube-api-access-5c4lb:{mountpoint:/var/lib/kubelet/pods/67c4f288-6571-456f-8aa8-d6f5947eb14d/volumes/kubernetes.io~projected/kube-api-access-5c4lb major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e/volumes/kubernetes.io~projected/kube-api-access-4kjjq:{mountpoint:/var/lib/kubelet/pods/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e/volumes/kubernetes.io~projected/kube-api-access-4kjjq major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c707771-1798-4442-9277-cecbc002c007/volumes/kubernetes.io~projected/kube-api-access-nxqdw:{mountpoint:/var/lib/kubelet/pods/6c707771-1798-4442-9277-cecbc002c007/volumes/kubernetes.io~projected/kube-api-access-nxqdw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7533bb1d-7252-4b45-b915-969a3f216ba4/volumes/kubernetes.io~projected/kube-api-access-svcv2:{mountpoint:/var/lib/kubelet/pods/7533bb1d-7252-4b45-b915-969a3f216ba4/volumes/kubernetes.io~projected/kube-api-access-svcv2 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/756d759c-3587-4ff9-9abb-f24c46f64077/volumes/kubernetes.io~projected/kube-api-access-ntq5r:{mountpoint:/var/lib/kubelet/pods/756d759c-3587-4ff9-9abb-f24c46f64077/volumes/kubernetes.io~projected/kube-api-access-ntq5r major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~projected/kube-api-access-hjd2s:{mountpoint:/var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~projected/kube-api-access-hjd2s major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volumes/kubernetes.io~projected/kube-api-access-plwpf:{mountpoint:/var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volumes/kubernetes.io~projected/kube-api-access-plwpf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dfd0ee8-e42a-4bbd-b355-44262861da53/volumes/kubernetes.io~projected/kube-api-access-dcvv7:{mountpoint:/var/lib/kubelet/pods/8dfd0ee8-e42a-4bbd-b355-44262861da53/volumes/kubernetes.io~projected/kube-api-access-dcvv7 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90d658ad-d248-4853-8bd8-b64076ec9968/volumes/kubernetes.io~projected/kube-api-access-mwrjl:{mountpoint:/var/lib/kubelet/pods/90d658ad-d248-4853-8bd8-b64076ec9968/volumes/kubernetes.io~projected/kube-api-access-mwrjl major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9779be5d-063f-4f06-8e58-746db277ce66/volumes/kubernetes.io~projected/kube-api-access-vfcqf:{mountpoint:/var/lib/kubelet/pods/9779be5d-063f-4f06-8e58-746db277ce66/volumes/kubernetes.io~projected/kube-api-access-vfcqf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a36a016-71e1-4a23-b4ca-768e186091e3/volumes/kubernetes.io~projected/kube-api-access-46qm8:{mountpoint:/var/lib/kubelet/pods/9a36a016-71e1-4a23-b4ca-768e186091e3/volumes/kubernetes.io~projected/kube-api-access-46qm8 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5/volumes/kubernetes.io~projected/kube-api-access-dr2lq:{mountpoint:/var/lib/kubelet/pods/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5/volumes/kubernetes.io~projected/kube-api-access-dr2lq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~projected/kube-api-access-b4t5z:{mountpoint:/var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~projected/kube-api-access-b4t5z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad03e4e-7139-463f-baf7-752544c6763f/volumes/kubernetes.io~projected/kube-api-access-bvqq7:{mountpoint:/var/lib/kubelet/pods/aad03e4e-7139-463f-baf7-752544c6763f/volumes/kubernetes.io~projected/kube-api-access-bvqq7 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad03e4e-7139-463f-baf7-752544c6763f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/aad03e4e-7139-463f-baf7-752544c6763f/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06c9464-bdf9-493a-8d44-37cca42409d1/volumes/kubernetes.io~projected/kube-api-access-6k8jn:{mountpoint:/var/lib/kubelet/pods/b06c9464-bdf9-493a-8d44-37cca42409d1/volumes/kubernetes.io~projected/kube-api-access-6k8jn major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06c9464-bdf9-493a-8d44-37cca42409d1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b06c9464-bdf9-493a-8d44-37cca42409d1/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8edf410-4aa2-42ea-bf3f-016002b74ffa/volumes/kubernetes.io~projected/kube-api-access-8t9qc:{mountpoint:/var/lib/kubelet/pods/b8edf410-4aa2-42ea-bf3f-016002b74ffa/volumes/kubernetes.io~projected/kube-api-access-8t9qc major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8c42546-d756-4965-96c3-e3e05dd3d89b/volumes/kubernetes.io~projected/kube-api-access-5nwwt:{mountpoint:/var/lib/kubelet/pods/c8c42546-d756-4965-96c3-e3e05dd3d89b/volumes/kubernetes.io~projected/kube-api-access-5nwwt major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8f7392b-faf8-43ad-a96b-7e95a95d7792/volumes/kubernetes.io~projected/kube-api-access-dczl4:{mountpoint:/var/lib/kubelet/pods/c8f7392b-faf8-43ad-a96b-7e95a95d7792/volumes/kubernetes.io~projected/kube-api-access-dczl4 major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce8955fd-a862-461a-bce9-a5be688358ef/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ce8955fd-a862-461a-bce9-a5be688358ef/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4043168-4d17-4609-bc2f-37951b7480b4/volumes/kubernetes.io~projected/kube-api-access-kfklt:{mountpoint:/var/lib/kubelet/pods/e4043168-4d17-4609-bc2f-37951b7480b4/volumes/kubernetes.io~projected/kube-api-access-kfklt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c6a49a-2e5b-460c-8c6a-ca303019575a/volumes/kubernetes.io~projected/kube-api-access-x5gll:{mountpoint:/var/lib/kubelet/pods/f3c6a49a-2e5b-460c-8c6a-ca303019575a/volumes/kubernetes.io~projected/kube-api-access-x5gll major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9191bfb-6d65-4d74-8948-fcdb9ad57b76/volumes/kubernetes.io~projected/kube-api-access-7fsgl:{mountpoint:/var/lib/kubelet/pods/f9191bfb-6d65-4d74-8948-fcdb9ad57b76/volumes/kubernetes.io~projected/kube-api-access-7fsgl major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9191bfb-6d65-4d74-8948-fcdb9ad57b76/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f9191bfb-6d65-4d74-8948-fcdb9ad57b76/volumes/kubernetes.io~secret/cert major:0 minor:211 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/146beac159e59f857528241a923cdd277b9688d34a1325fc8a1be5cfe3dd1c8a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/49ea83d1ea9169530218915a5676877806c100d63cf124b23a252ba56217aeb8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6d673d903d07bf2b5f936ae16feac23ae3b2d0feec4808a810c38808b394a049/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d7d41792432764b0756792962bf0848c3e3d9c8547003826bdd78851c1fe8c86/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9c6a701a6d1afc286a4ec76c3a654900938710819e1e9243ed79d33f187e045c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7e9169b92994b737de5aaa4b48a0b479232f889cb151365053f775a597418c71/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d8ccc6b349cb68e79e9c090f8e139e423c2d8d484515ac51f04fb1d2c33ef965/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/caaa80ff574eae06ba85a70e0b968bc7d5827cf2a94a4c125e7d2df4a5cca75d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3d3bff8474757c578064384eab8087c4bb94011666d8844c10d6141b21557ed2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/4c7567502da0a35e17136934bdb5a034578190f9875320786032282fbe78d38e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/54105d8e4dc8c8da36cb321e59c1228ae2ad6be47eb7666b066969c3eb913ee4/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/8b8c562a03ffeddccd7475da3f15f22e5690bf6ca795a1cd8f268e616881ddd1/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/118553b56496c8bc700d3bb067030dfd1eef18c497b4913525010dcaebf75a43/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a8a740c6e32191f0ea76a8dbc825e13182a7614332552a0fa7233ebe955a84ac/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/c534133fa6c3e798bda9ab8c6f497232120274d4a2200aff418512b524c07c9b/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/52fc4255aa4312e17ad6b5863b6f9fe3fe00142fbc79000f97498472daf83811/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/442adf3757eb4a9e3f1ded6914538ace595ded6612a0cc92dad090ee25195160/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a489e6da7e061c07fc73b7d2ebcfb22f8ba3b398b4c6aba2407dfd8936edfc00/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1e2f3a3154a1c8edc20ea80521f57330bea38b96c7fb745a3c66fabb06ddf253/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/039c1985921e288f14fa0582f7bd06f6abb1111b31180c34a39736a4b924a312/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3218358199f5da5ae32adf90b80b83d0033d0439c85aa88c5775dbecbef70a36/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/13a1d0289874059d2f9f2a5a8a1c7927fe6d6442c8a05432cc7b3ea62ad3ac6e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b0643f17bcf031d5f5e0318417305373fe5a2145fd538366a04153a4ee83f8aa/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/18bae38f4f9d9da81d366c9528fc85a1f226d90be50188909b11dd844f445ecb/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/8ae55f810dd9d241f9b2438bc1bb8fa1981ee30ee18f36dd2ff5528f9f53d38d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9f25a00d5fd9f4dc2e170c304062c908b552bb61551cc1da608c1568b96c8013/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/bd8ab678fb4e250fd1c2d27610a280959ed1dae3c0cf339c6c9222a79f86dc67/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/00c6905b3898b1f788bc1f5f6fb029b1c0a4decd971c65cbc7d727628dde6f9d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b71408d248337b60e2efca2bd04a95ddcdee95ea56ca929d4097a5ca4f4c9025/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/52d9358260552fa578c7f2968884c6f1edee17d539594a9975aa5072f726c54b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/313867d22d552302602e5497644ad9936924aacb9d9f45830e600d9d4a077443/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/5a58c248a1d53c79a5cc08fab36d68f3c15266dc65fb43306a841368465748c4/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/c9bef63add16b4b548f69fec9b004e9d0a759eb5e0bb7e25e545efe0f63ebf14/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/65978178b6594edd6b239fd17b78efe754af81b965579a5d5a12da63062b6749/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/181c10dbfb79d8d98aa548040d5a4d59f78bb6371a634b1b8f23ab898454cce1/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/1d4232d44a8f3ca4c06503994730c6c3dda0bc68482e7ccce1f79851b35dfb5a/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/61f8a86699801bd8a132780242f04e35a585a5a1acc6c83af096ad7234017fb4/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/2fbb722fdd3f5970a56485eebd4b848d7b40ef2b67e73d318430573b19793882/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6b07efdb97834d53164fb446057f1559ef9a69a89ebbd90260cd3ec0957924d9/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/81cd6ee5d3edb1808cabc00a694cba522a9d8d6c0af21f409338308e0627452d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/e6432339f78fba08e4a43fb5092784f9371a7c23e3dad34a8bfa46602ad3e0ed/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/ace5c054925273f5dc8469dcbb22ed8d35d12e777ef6bde5660cf89f777e0f0d/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/032834456ac3243272315882f8940d7b2695b68d2de771f5a31775c0dc5fd41a/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/8ae1520573a1bba032be209ce2eaf07179097dec39207383878c2ff4e62458bb/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/f1c0c9c30d39fd4c8cd3fd03477056be0c3a238f56d030feec2d864678b4a648/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/ea4c21ffcc36a494b9680ea62b2547e6b4a30eadb35e334ff82f8038d2ac903c/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/6a72c3ddaa5e276a8ee6090cc677ba8ae3f545e315ca95bd3eee5b98f7e4b4d5/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/95c01fb5e39c8cd6691774bf2240f8e5419cfbf0e7d7366399caa9f3591e2240/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/98330edf40ed7d40f403283ea7beb7c3fc27493737afd0bd617d2650f8803f2e/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/c327292a7ee3ddf9914749a3378e9e0f92c75ceeac6df68311f77964c2b32776/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/5ddb7a363ed96453f3ec8d6bf8c8b850f8ca4ba0b318f5dbe5d33224d55c4596/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/ec763c5e25f331530b8653833300e36854e7b110a109bb2dc3c4c4ef5e2244be/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1f397893ffa4edddbe8512fa011ef545d8e87bb7bc369fcaa2bdb71eb10ca29f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7bf8974a25e4a4a4a1ad50e7370805480afacbd99010f73d25bccc0e04422815/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/004c8919bb82b3c3b3e0376bcedfd7315ea98860d4c52bce0a6a1eb38ec8b1c8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/dca75575997a043f2406e5f9ff56018fbfb8850f93735fe450a54a2a68d83d01/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/4134a9cb0109ee4fc5c1b9a785e78c958caffeb16014880fd61e4361a9265aee/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/00bd69333f5ef8afb445406750acca9caee1091559e0872e2b7a7613f80d093b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/f6472d8d9e7cb9242e44b4200845964fb6f5046d8582eb423b34a7f242ec3c7f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/ee63ad6dd46bc943120043c4150f8f3dc87ecd344fdc96b6ddcea239217c4fc9/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6cb1758f82aeb54dddb19aec4e52f23f5705b6fcdf8ad451e19491b690f6ff98/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c1f415e2ebf97b8df8ddd9afce820896e1507b30d6b718d6ab8820759aed912e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 22:49:33.875758 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.875296 5043 manager.go:217] Machine: {Timestamp:2026-10-31 22:49:33.874481263 +0000 UTC m=+0.104035670 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099780 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec285524684e78a4163432431ad8a1a8 SystemUUID:ec285524-684e-78a4-1634-32431ad8a1a8 BootID:4adba19c-fefa-4c74-8808-bde9684fe012 Filesystems:[{Device:/run/containers/storage/overlay-containers/1a575886f029be42493e69eed5bc90068c08493a41224d490467fe6542f9ef89/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~projected/kube-api-access-wzfqz DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~projected/kube-api-access-b4t5z DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88a83900951ee18ae1b9dbd0ae556bcda488684b10acbba950bf6bfeced490cf/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb715412f9c1cbcc21f60d5e76e2ffcd1be2092a59043d972e9d63b4c169042f/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9191bfb-6d65-4d74-8948-fcdb9ad57b76/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90d658ad-d248-4853-8bd8-b64076ec9968/volumes/kubernetes.io~projected/kube-api-access-mwrjl DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8e97e6e47247fd85485f13a5fd5ea48eb2498650775a449a655d1458302076f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6f52986c77878b7381a1aea61289566f0eb3e204ccfd21b8fc659b489cc8d1d/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3b8a5ea7eabbce76dc72f1e3d7a7c5c09edfdb2f50b21925982a7b59668915b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d78eab4961209bc6570ee16be900f3c54e0c9d9ba278c31407f5327f6ef370b/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a36a016-71e1-4a23-b4ca-768e186091e3/volumes/kubernetes.io~projected/kube-api-access-46qm8 DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8f7392b-faf8-43ad-a96b-7e95a95d7792/volumes/kubernetes.io~projected/kube-api-access-dczl4 DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/282c0fbadd0dd20a6dda66afea75c770e7c8031d732594a78d743151bb23b842/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5911dd1863e5a0316aa952e6431d0b43139a6dc21a3b30a4c3822e3f0dc4e963/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b4ccbacdf0b2279bcf7d401945b1b609b5f85365bcb92ac160dbc59172acb4f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aad03e4e-7139-463f-baf7-752544c6763f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a940d2c-fc2c-40ae-a8bd-5916089eb925/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4ab4019b4720c5bc77f16491589034547295df822f1d476f941024b8fd344f8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01689403-7698-4f38-b4c7-8a3603742ca8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8dfd0ee8-e42a-4bbd-b355-44262861da53/volumes/kubernetes.io~projected/kube-api-access-dcvv7 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volumes/kubernetes.io~projected/kube-api-access-plwpf DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca0505a82716dd34b05601d7ea866db3288c6a9f4c0af1e39ee34ad5f648a5ba/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9779be5d-063f-4f06-8e58-746db277ce66/volumes/kubernetes.io~projected/kube-api-access-vfcqf DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aad03e4e-7139-463f-baf7-752544c6763f/volumes/kubernetes.io~projected/kube-api-access-bvqq7 DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdda817e3fa1a3245b16866e9463753289b7eaf62944ef4af644f8d2ad70411d/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/756d759c-3587-4ff9-9abb-f24c46f64077/volumes/kubernetes.io~projected/kube-api-access-ntq5r DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce8955fd-a862-461a-bce9-a5be688358ef/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4043168-4d17-4609-bc2f-37951b7480b4/volumes/kubernetes.io~projected/kube-api-access-kfklt DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce54792-3b37-4895-a39a-d233a7e3d4ee/volumes/kubernetes.io~projected/kube-api-access-q4bjc DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7533bb1d-7252-4b45-b915-969a3f216ba4/volumes/kubernetes.io~projected/kube-api-access-svcv2 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e/volumes/kubernetes.io~projected/kube-api-access-4kjjq DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9/volumes/kubernetes.io~projected/kube-api-access-2kmq4 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/volumes/kubernetes.io~projected/kube-api-access-nvgzm DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9191bfb-6d65-4d74-8948-fcdb9ad57b76/volumes/kubernetes.io~projected/kube-api-access-7fsgl DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b06c9464-bdf9-493a-8d44-37cca42409d1/volumes/kubernetes.io~projected/kube-api-access-6k8jn DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8edf410-4aa2-42ea-bf3f-016002b74ffa/volumes/kubernetes.io~projected/kube-api-access-8t9qc DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d95fc77-4bb6-4464-97f2-2bd5500d88e1/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5/volumes/kubernetes.io~projected/kube-api-access-dr2lq DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d2027bda29a09cbeb1e5dae5fb52a0ceee1a0178ce96bbe6370b10a614a884f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39a8980d-f812-4330-84e7-c406c8afa702/volumes/kubernetes.io~projected/kube-api-access-kc5c8 DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13e8e748-f088-43d1-8011-bf6c8f91ec59/volumes/kubernetes.io~projected/kube-api-access-nsnkd DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/001f1850f331e34b558250e9a8118ed7315d5ae31dfd157a90d55a89306c3641/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67c4f288-6571-456f-8aa8-d6f5947eb14d/volumes/kubernetes.io~projected/kube-api-access-5c4lb DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c6a49a-2e5b-460c-8c6a-ca303019575a/volumes/kubernetes.io~projected/kube-api-access-x5gll DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8c42546-d756-4965-96c3-e3e05dd3d89b/volumes/kubernetes.io~projected/kube-api-access-5nwwt DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7afda5df-8528-4304-aa1d-156aefde0156/volumes/kubernetes.io~projected/kube-api-access-hjd2s DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3916f8cff62a119277f838c18cf52667cfc9aac6b9fc12a1bcd0f620e64715c6/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/482c6b59ce34a880dd953eed040fa85bc06265e9cfbf941c44da850d44fe34ae/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ecbeedd52b8b63a6eeb09fa7225d8a48a0b2e7261ff643b1bea303ba5291a7d/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25beed3c7189cf2f4927c3d3486423abab05ff0cf4ddde52830875b5f83feba3/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a1ca4ab-6864-49ea-af42-3c31f7823285/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~projected/kube-api-access-plbl9 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d95fc77-4bb6-4464-97f2-2bd5500d88e1/volumes/kubernetes.io~projected/kube-api-access-nxjq8 DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c707771-1798-4442-9277-cecbc002c007/volumes/kubernetes.io~projected/kube-api-access-nxqdw DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01689403-7698-4f38-b4c7-8a3603742ca8/volumes/kubernetes.io~projected/kube-api-access-gdlrr DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6a682f254cba7e584cab6af87e32004405157f3a8a386d5d9fee97aff00663/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b864f135b25bcca53bc9f4eb4b48b768b13907d13d9722ddf4e826e9bc250f1/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37b0f64a-6235-4472-b7e2-a2d2a08fd388/volumes/kubernetes.io~projected/kube-api-access-g9868 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b06c9464-bdf9-493a-8d44-37cca42409d1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84a1f1f411a690fd407391e0a2cf7cb3bfed147bb80eb4db185832a2740e7dcf/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00892423acc13729a96de605330aa4ad0716f14510f8bd832664d5ddf59b7a48/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00892423acc1372 MacAddress:66:ae:49:43:18:ea Speed:10000 Mtu:8901} {Name:25beed3c7189cf2 MacAddress:3a:68:c6:c1:02:1d Speed:10000 Mtu:8901} {Name:282c0fbadd0dd20 MacAddress:fe:69:53:50:91:a7 Speed:10000 Mtu:8901} {Name:3916f8cff62a119 MacAddress:b2:af:9a:bd:09:4a Speed:10000 Mtu:8901} {Name:482c6b59ce34a88 MacAddress:76:4e:b9:a1:83:ac Speed:10000 Mtu:8901} {Name:4b4ccbacdf0b227 MacAddress:22:7d:84:cb:d6:5e Speed:10000 Mtu:8901} {Name:4b864f135b25bcc MacAddress:16:4b:1f:a7:29:f8 Speed:10000 Mtu:8901} {Name:5911dd1863e5a03 MacAddress:86:d1:db:ac:16:9e Speed:10000 Mtu:8901} {Name:84a1f1f411a690f MacAddress:22:b5:1c:bc:0a:c7 Speed:10000 Mtu:8901} {Name:88a83900951ee18 MacAddress:5e:4d:90:2b:67:da Speed:10000 Mtu:8901} {Name:8d78eab4961209b MacAddress:b6:8a:18:f5:39:76 Speed:10000 Mtu:8901} {Name:8ecbeedd52b8b63 MacAddress:9e:d7:2a:30:b8:12 Speed:10000 Mtu:8901} {Name:a0bb1f3df96c1d1 MacAddress:6e:4d:e7:57:85:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:f1:92:bf Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:b9:b6:01:f0:8c Speed:0 Mtu:8901} {Name:d6f52986c77878b MacAddress:a6:41:39:14:3b:fc Speed:10000 Mtu:8901} {Name:e8e97e6e47247fd MacAddress:d2:33:0e:d9:05:a7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:f1:92:bf Speed:0 Mtu:9001} {Name:fdda817e3fa1a32 MacAddress:4e:ac:85:98:3b:91 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:32:3d:2f:de:19:65 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:24:a1:3a:b6:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 22:49:33.876040 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.875766 5043 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 22:49:33.876040 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.875828 5043 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 22:49:33.876135 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876114 5043 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 22:49:33.876253 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876136 5043 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-222.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 22:49:33.876297 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876254 5043 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 22:49:33.876297 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876262 5043 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 22:49:33.876297 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876272 5043 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:49:33.876297 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876284 5043 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 22:49:33.876394 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876303 5043 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 22:49:33.876449 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876439 5043 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 22:49:33.876557 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876535 5043 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 22:49:33.876592 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876582 5043 kubelet.go:493] "Attempting to sync node with API server" Oct 31 22:49:33.876620 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876594 5043 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 22:49:33.876620 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876604 5043 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 22:49:33.876620 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876612 5043 kubelet.go:405] "Adding apiserver pod source" Oct 31 22:49:33.876620 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.876620 5043 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 22:49:33.877390 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.877373 5043 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 22:49:33.877390 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.877389 5043 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:49:33.877911 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.877894 5043 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 22:49:33.880713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.880692 5043 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 22:49:33.880713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.880712 5043 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 22:49:33.880863 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.880839 5043 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 22:49:33.881048 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881036 5043 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 22:49:33.881098 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881066 5043 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 22:49:33.881249 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881238 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 22:49:33.881283 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881261 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 22:49:33.881283 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881275 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 22:49:33.881335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881286 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 22:49:33.881335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881294 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 22:49:33.881335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881304 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 22:49:33.881419 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881338 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 22:49:33.881419 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881350 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 22:49:33.881419 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881360 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 22:49:33.881419 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881367 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 22:49:33.881419 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881387 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 22:49:33.881419 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881398 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 22:49:33.881600 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881422 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 22:49:33.881600 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.881430 5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 22:49:33.885070 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.885055 5043 server.go:1263] "Started kubelet" Oct 31 22:49:33.885164 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.885136 5043 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 22:49:33.885811 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.885627 5043 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 22:49:33.885811 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.885725 5043 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 22:49:33.885845 ip-10-0-133-222 systemd[1]: Started Kubernetes Kubelet. Oct 31 22:49:33.886064 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.886047 5043 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 22:49:33.890526 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.887793 5043 server.go:317] "Adding debug handlers to kubelet server" Oct 31 22:49:33.892829 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.892809 5043 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 22:49:33.892829 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.892824 5043 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 22:49:33.892962 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.892931 5043 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 22:49:33.892962 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.892943 5043 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 22:49:33.893065 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.892964 5043 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 22:43:53 +0000 UTC" deadline="2028-05-17 00:28:19.621560263 +0000 UTC" Oct 31 22:49:33.893065 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.892999 5043 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13513h38m45.728564418s" Oct 31 22:49:33.893065 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:33.893033 5043 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.us-east-2.compute.internal\" not found" Oct 31 22:49:33.893065 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.893048 5043 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 22:49:33.895055 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895034 5043 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895066 5043 factory.go:55] Registering systemd factory Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895076 5043 factory.go:223] Registration of the systemd container factory successfully Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895526 5043 factory.go:153] Registering CRI-O factory Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895556 5043 factory.go:223] Registration of the crio container factory successfully Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895580 5043 factory.go:103] Registering Raw factory Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895594 5043 manager.go:1199] Started watching for new ooms in manager Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.895970 5043 manager.go:319] Starting recovery of all containers Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896138 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a36a016-71e1-4a23-b4ca-768e186091e3" volumeName="kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896170 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad03e4e-7139-463f-baf7-752544c6763f" volumeName="kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-service-ca-bundle" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896183 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896196 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a940d2c-fc2c-40ae-a8bd-5916089eb925" volumeName="kubernetes.io/secret/5a940d2c-fc2c-40ae-a8bd-5916089eb925-agent-certs" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896210 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90d658ad-d248-4853-8bd8-b64076ec9968" volumeName="kubernetes.io/projected/90d658ad-d248-4853-8bd8-b64076ec9968-kube-api-access-mwrjl" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896221 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3" volumeName="kubernetes.io/projected/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-kube-api-access-b4t5z" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896232 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad03e4e-7139-463f-baf7-752544c6763f" volumeName="kubernetes.io/secret/aad03e4e-7139-463f-baf7-752544c6763f-serving-cert" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896244 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06c9464-bdf9-493a-8d44-37cca42409d1" volumeName="kubernetes.io/configmap/b06c9464-bdf9-493a-8d44-37cca42409d1-config" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896258 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c6a49a-2e5b-460c-8c6a-ca303019575a" volumeName="kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cni-binary-copy" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896270 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9191bfb-6d65-4d74-8948-fcdb9ad57b76" volumeName="kubernetes.io/projected/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-kube-api-access-7fsgl" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896284 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d95fc77-4bb6-4464-97f2-2bd5500d88e1" volumeName="kubernetes.io/configmap/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-cabundle" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896296 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37b0f64a-6235-4472-b7e2-a2d2a08fd388" volumeName="kubernetes.io/configmap/37b0f64a-6235-4472-b7e2-a2d2a08fd388-serviceca" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896309 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1ca4ab-6864-49ea-af42-3c31f7823285" volumeName="kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-script-lib" seLinuxMountContext="" Oct 31 22:49:33.896335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896322 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a36a016-71e1-4a23-b4ca-768e186091e3" volumeName="kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896360 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dfd0ee8-e42a-4bbd-b355-44262861da53" volumeName="kubernetes.io/projected/8dfd0ee8-e42a-4bbd-b355-44262861da53-kube-api-access-dcvv7" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896374 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a36a016-71e1-4a23-b4ca-768e186091e3" volumeName="kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896386 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" volumeName="kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896398 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4043168-4d17-4609-bc2f-37951b7480b4" volumeName="kubernetes.io/projected/e4043168-4d17-4609-bc2f-37951b7480b4-kube-api-access-kfklt" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896409 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" volumeName="kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-config" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896421 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" volumeName="kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-trusted-ca" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896433 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d1c440e-90c0-4411-84a4-a5bfaf91d3e9" volumeName="kubernetes.io/projected/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9-kube-api-access-2kmq4" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896444 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7afda5df-8528-4304-aa1d-156aefde0156" volumeName="kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-stats-auth" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896456 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1ca4ab-6864-49ea-af42-3c31f7823285" volumeName="kubernetes.io/secret/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896468 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" volumeName="kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896479 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3" volumeName="kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-tuned" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896500 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c6a49a-2e5b-460c-8c6a-ca303019575a" volumeName="kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-daemon-config" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896513 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896524 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896593 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a36a016-71e1-4a23-b4ca-768e186091e3" volumeName="kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896610 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896624 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce54792-3b37-4895-a39a-d233a7e3d4ee" volumeName="kubernetes.io/projected/3ce54792-3b37-4895-a39a-d233a7e3d4ee-kube-api-access-q4bjc" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896636 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a940d2c-fc2c-40ae-a8bd-5916089eb925" volumeName="kubernetes.io/configmap/5a940d2c-fc2c-40ae-a8bd-5916089eb925-konnectivity-ca" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896656 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67c4f288-6571-456f-8aa8-d6f5947eb14d" volumeName="kubernetes.io/configmap/67c4f288-6571-456f-8aa8-d6f5947eb14d-config-volume" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896671 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1ca4ab-6864-49ea-af42-3c31f7823285" volumeName="kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-env-overrides" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896684 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1ca4ab-6864-49ea-af42-3c31f7823285" volumeName="kubernetes.io/projected/8a1ca4ab-6864-49ea-af42-3c31f7823285-kube-api-access-plwpf" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896697 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" volumeName="kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-binary-copy" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896709 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad03e4e-7139-463f-baf7-752544c6763f" volumeName="kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-tmp" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896723 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13e8e748-f088-43d1-8011-bf6c8f91ec59" volumeName="kubernetes.io/empty-dir/13e8e748-f088-43d1-8011-bf6c8f91ec59-tmp-dir" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896735 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67c4f288-6571-456f-8aa8-d6f5947eb14d" volumeName="kubernetes.io/projected/67c4f288-6571-456f-8aa8-d6f5947eb14d-kube-api-access-5c4lb" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896747 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="756d759c-3587-4ff9-9abb-f24c46f64077" volumeName="kubernetes.io/empty-dir/756d759c-3587-4ff9-9abb-f24c46f64077-data-volume" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896761 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9779be5d-063f-4f06-8e58-746db277ce66" volumeName="kubernetes.io/configmap/9779be5d-063f-4f06-8e58-746db277ce66-iptables-alerter-script" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896792 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad03e4e-7139-463f-baf7-752544c6763f" volumeName="kubernetes.io/projected/aad03e4e-7139-463f-baf7-752544c6763f-kube-api-access-bvqq7" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896806 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06c9464-bdf9-493a-8d44-37cca42409d1" volumeName="kubernetes.io/projected/b06c9464-bdf9-493a-8d44-37cca42409d1-kube-api-access-6k8jn" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896825 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eec52f11-3785-48d2-92a1-cbd30a6047e2" volumeName="kubernetes.io/configmap/eec52f11-3785-48d2-92a1-cbd30a6047e2-nginx-conf" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896846 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01689403-7698-4f38-b4c7-8a3603742ca8" volumeName="kubernetes.io/projected/01689403-7698-4f38-b4c7-8a3603742ca8-kube-api-access-gdlrr" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896857 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a1ca4ab-6864-49ea-af42-3c31f7823285" volumeName="kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-config" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896881 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8c42546-d756-4965-96c3-e3e05dd3d89b" volumeName="kubernetes.io/projected/c8c42546-d756-4965-96c3-e3e05dd3d89b-kube-api-access-5nwwt" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896894 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9191bfb-6d65-4d74-8948-fcdb9ad57b76" volumeName="kubernetes.io/secret/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-cert" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896908 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896920 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896931 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d95fc77-4bb6-4464-97f2-2bd5500d88e1" volumeName="kubernetes.io/projected/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-kube-api-access-nxjq8" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896944 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896957 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8955fd-a862-461a-bce9-a5be688358ef" volumeName="kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896969 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" volumeName="kubernetes.io/projected/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-kube-api-access-dr2lq" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.896989 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad03e4e-7139-463f-baf7-752544c6763f" volumeName="kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-trusted-ca-bundle" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897001 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06c9464-bdf9-493a-8d44-37cca42409d1" volumeName="kubernetes.io/secret/b06c9464-bdf9-493a-8d44-37cca42409d1-serving-cert" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897021 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8edf410-4aa2-42ea-bf3f-016002b74ffa" volumeName="kubernetes.io/configmap/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-config" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897032 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8f7392b-faf8-43ad-a96b-7e95a95d7792" volumeName="kubernetes.io/projected/c8f7392b-faf8-43ad-a96b-7e95a95d7792-kube-api-access-dczl4" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897054 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897066 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897077 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897098 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37b0f64a-6235-4472-b7e2-a2d2a08fd388" volumeName="kubernetes.io/projected/37b0f64a-6235-4472-b7e2-a2d2a08fd388-kube-api-access-g9868" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897111 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a8980d-f812-4330-84e7-c406c8afa702" volumeName="kubernetes.io/projected/39a8980d-f812-4330-84e7-c406c8afa702-kube-api-access-kc5c8" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897124 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c707771-1798-4442-9277-cecbc002c007" volumeName="kubernetes.io/projected/6c707771-1798-4442-9277-cecbc002c007-kube-api-access-nxqdw" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897139 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13e8e748-f088-43d1-8011-bf6c8f91ec59" volumeName="kubernetes.io/projected/13e8e748-f088-43d1-8011-bf6c8f91ec59-kube-api-access-nsnkd" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897150 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7533bb1d-7252-4b45-b915-969a3f216ba4" volumeName="kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897164 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" volumeName="kubernetes.io/secret/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-serving-cert" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897177 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897188 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c14a7d4-f9a1-4e58-8ee5-8455aecf175e" volumeName="kubernetes.io/projected/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e-kube-api-access-4kjjq" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897217 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7afda5df-8528-4304-aa1d-156aefde0156" volumeName="kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-default-certificate" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897229 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad03e4e-7139-463f-baf7-752544c6763f" volumeName="kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-snapshots" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897241 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8edf410-4aa2-42ea-bf3f-016002b74ffa" volumeName="kubernetes.io/projected/b8edf410-4aa2-42ea-bf3f-016002b74ffa-kube-api-access-8t9qc" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897254 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7afda5df-8528-4304-aa1d-156aefde0156" volumeName="kubernetes.io/projected/7afda5df-8528-4304-aa1d-156aefde0156-kube-api-access-hjd2s" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897265 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01689403-7698-4f38-b4c7-8a3603742ca8" volumeName="kubernetes.io/secret/01689403-7698-4f38-b4c7-8a3603742ca8-serving-cert" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897291 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67c4f288-6571-456f-8aa8-d6f5947eb14d" volumeName="kubernetes.io/empty-dir/67c4f288-6571-456f-8aa8-d6f5947eb14d-tmp-dir" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897303 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dfd0ee8-e42a-4bbd-b355-44262861da53" volumeName="kubernetes.io/configmap/8dfd0ee8-e42a-4bbd-b355-44262861da53-telemetry-config" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897345 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9779be5d-063f-4f06-8e58-746db277ce66" volumeName="kubernetes.io/projected/9779be5d-063f-4f06-8e58-746db277ce66-kube-api-access-vfcqf" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897359 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c6a49a-2e5b-460c-8c6a-ca303019575a" volumeName="kubernetes.io/projected/f3c6a49a-2e5b-460c-8c6a-ca303019575a-kube-api-access-x5gll" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897372 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d95fc77-4bb6-4464-97f2-2bd5500d88e1" volumeName="kubernetes.io/secret/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-key" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897384 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897396 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3" volumeName="kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-tmp" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897408 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" volumeName="kubernetes.io/projected/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-kube-api-access-nvgzm" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897421 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01689403-7698-4f38-b4c7-8a3603742ca8" volumeName="kubernetes.io/configmap/01689403-7698-4f38-b4c7-8a3603742ca8-config" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897432 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec08e29-8387-4709-b902-87580b7cc86a" volumeName="kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897444 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e299d0d-850e-4a45-b03b-24071e0d3459" volumeName="kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897455 5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="756d759c-3587-4ff9-9abb-f24c46f64077" volumeName="kubernetes.io/projected/756d759c-3587-4ff9-9abb-f24c46f64077-kube-api-access-ntq5r" seLinuxMountContext="" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897468 5043 reconstruct.go:97] "Volume reconstruction finished" Oct 31 22:49:33.897513 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.897476 5043 reconciler.go:29] "Reconciler: start to sync state" Oct 31 22:49:33.901903 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.901883 5043 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:33.903755 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.903728 5043 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:33.903918 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.903895 5043 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:33.905259 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.905239 5043 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 22:49:33.917775 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.917756 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:49:33.917894 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.917799 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:49:33.925815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.925782 5043 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 22:49:33.927384 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.927202 5043 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 22:49:33.927613 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.927592 5043 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 22:49:33.927688 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.927633 5043 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 22:49:33.927758 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:33.927679 5043 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 22:49:33.931953 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.931931 5043 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 22:49:33.943982 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.943958 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="7a7580bf8f4de6f8b2edc4479db5f361d3a272f910499c7357cfa25232062a74" exitCode=0 Oct 31 22:49:33.943982 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.943979 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="7454988180ec6e403d3847ad2bb0d4d2b1e628fd4f035333755ba46b139bbb16" exitCode=0 Oct 31 22:49:33.943982 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.943985 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="c06a7f6811c3e37774521e5dd901bd210df6c0731713ae210c2e660be65fc2cb" exitCode=0 Oct 31 22:49:33.944172 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.943990 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="4fc9f98684fce4cfba96e3236dea3e2613d9faf4c092ef13d546adfd1d74c99b" exitCode=0 Oct 31 22:49:33.944172 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.943996 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="647a55ba56e9e838475c9b0d014f61c9f50ab3170e930657739524b9437c1d94" exitCode=0 Oct 31 22:49:33.944172 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.944001 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ffdc8b2-d568-4e04-a6a8-681de62c2ab5" containerID="60a91d0230f9fd80b277dbf00d216adefa76b505abfe57d2a7b23dc96ba343f7" exitCode=0 Oct 31 22:49:33.970197 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.970139 5043 generic.go:356] "Generic (PLEG): container finished" podID="c87261ed136f2c9536dc41bf58fda7ff" containerID="18a893e7134d7ffb74b9587b0932c2d49f340649a1a3aa252f0eb29b4a867928" exitCode=0 Oct 31 22:49:33.975915 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.975895 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:49:33.975993 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.975925 5043 generic.go:356] "Generic (PLEG): container finished" podID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" exitCode=255 Oct 31 22:49:33.996747 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:33.996731 5043 manager.go:324] Recovery completed Oct 31 22:49:34.017411 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017393 5043 cpu_manager.go:226] "Starting" policy="none" Oct 31 22:49:34.017411 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017408 5043 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 22:49:34.017507 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017425 5043 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 22:49:34.017596 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017575 5043 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 22:49:34.017699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017592 5043 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 22:49:34.017699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017614 5043 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 22:49:34.017699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017628 5043 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 22:49:34.017699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017635 5043 policy_none.go:50] "Start" Oct 31 22:49:34.017699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017648 5043 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 22:49:34.017699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.017663 5043 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 22:49:34.018410 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.018390 5043 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 22:49:34.018410 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.018410 5043 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 22:49:34.018525 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.018419 5043 policy_none.go:44] "Start" Oct 31 22:49:34.023211 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.023187 5043 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 22:49:34.026744 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.026732 5043 manager.go:341] "Starting Device Plugin manager" Oct 31 22:49:34.026794 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:34.026754 5043 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 22:49:34.026794 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.026763 5043 server.go:87] "Starting device plugin registration server" Oct 31 22:49:34.027072 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027059 5043 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 22:49:34.027104 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027075 5043 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 22:49:34.027166 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027147 5043 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 22:49:34.027427 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027248 5043 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 22:49:34.027427 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027314 5043 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 22:49:34.027427 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027323 5043 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 22:49:34.027634 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.027392 5043 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T22:49:34.027280886Z","UUID":"1a264af8-bda6-44e2-a0fd-c69a721da59c","Handler":null,"Name":"","Endpoint":""} Oct 31 22:49:34.028082 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.028041 5043 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal"] Oct 31 22:49:34.028718 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:34.028700 5043 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 22:49:34.028865 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.028852 5043 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 22:49:34.028937 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.028873 5043 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 22:49:34.028937 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.028925 5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.029044 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.028984 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"a70adfcc128189ec0c0f585313e4a644","Type":"ContainerStarted","Data":"7b3ac75c44746ef6472efec9f018886048d7548d589b4dd94772ae3a2f1401a2"} Oct 31 22:49:34.029044 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.029031 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"a70adfcc128189ec0c0f585313e4a644","Type":"ContainerStarted","Data":"25425bbb9381ea89c69b7789a86ab7fe8c8b7a5006658923ca8ab9f532288894"} Oct 31 22:49:34.029141 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.029071 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"c87261ed136f2c9536dc41bf58fda7ff","Type":"ContainerStarted","Data":"cfe67ac2594be1a31066cc11d050ce0ed91aa3e517cf4d6b6a9cec53f4de0f38"} Oct 31 22:49:34.029141 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.029090 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"c87261ed136f2c9536dc41bf58fda7ff","Type":"ContainerDied","Data":"18a893e7134d7ffb74b9587b0932c2d49f340649a1a3aa252f0eb29b4a867928"} Oct 31 22:49:34.029141 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.029108 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" event={"ID":"c87261ed136f2c9536dc41bf58fda7ff","Type":"ContainerStarted","Data":"efb8a1af4866bf6c92b7c1b0c40667002e5acc35d96259e198b13ab63ad7ada9"} Oct 31 22:49:34.029343 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.029327 5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.045990 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.045967 5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:49:34.046071 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:34.046007 5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.046448 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.046432 5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:49:34.046534 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:34.046465 5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.106034 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.106007 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.106113 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.106037 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.106113 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.106056 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70adfcc128189ec0c0f585313e4a644-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"a70adfcc128189ec0c0f585313e4a644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.128123 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.128103 5043 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 22:49:34.129174 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.129159 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 22:49:34.129245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.129186 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 22:49:34.129245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.129196 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 22:49:34.129245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.129233 5043 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.144357 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.144339 5043 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.144446 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.144396 5043 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.206876 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.206855 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.206947 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.206813 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.206947 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.206919 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.206947 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.206936 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70adfcc128189ec0c0f585313e4a644-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"a70adfcc128189ec0c0f585313e4a644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.207047 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.206976 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70adfcc128189ec0c0f585313e4a644-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"a70adfcc128189ec0c0f585313e4a644\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.207047 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.206999 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c87261ed136f2c9536dc41bf58fda7ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" (UID: \"c87261ed136f2c9536dc41bf58fda7ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.877424 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.877394 5043 apiserver.go:52] "Watching apiserver" Oct 31 22:49:34.895214 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.895187 5043 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 22:49:34.895908 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.895884 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jpmrk","openshift-multus/network-metrics-daemon-cz8ch","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal","openshift-dns/dns-default-xctjg","kube-system/global-pull-secret-syncer-xrfs4","kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-f6f9f","openshift-insights/periodic-gathering-4xsrk-mwv8t","openshift-ingress/router-default-8f9bc98f7-xn5rp","openshift-image-registry/node-ca-58lh2","kube-system/kas-connection-checker-685895c7c9-khbjk","openshift-cluster-node-tuning-operator/tuned-gljgz","openshift-ingress-canary/ingress-canary-r2hnp","openshift-multus/multus-8d5v6","openshift-multus/multus-additional-cni-plugins-gbvzb","openshift-service-ca/service-ca-5fffc97bdf-8g88j","kube-system/kas-connection-checker-685895c7c9-k5t2f","openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w","openshift-image-registry/image-registry-786cddd9dc-dfvvd","openshift-network-operator/iptables-alerter-q85ln","openshift-insights/insights-runtime-extractor-wtb2p","openshift-console-operator/console-operator-7cd56b8dbc-vkldg","openshift-image-registry/image-registry-5c6987c8-t24sb","openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9","openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l","openshift-network-console/networking-console-plugin-86c478bb88-qv9l7","openshift-network-diagnostics/network-check-target-56mmn","openshift-dns/node-resolver-sk99x","openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688","openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9","kube-system/kas-connection-checker-685895c7c9-gftjj","openshift-insights/insights-operator-7c4986d449-fj2lt"] Oct 31 22:49:34.898022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.898000 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:34.898359 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.898338 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:34.899416 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.899397 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:34.900213 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.900197 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:34.900695 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.900679 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:34.902366 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.902340 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.902495 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.902479 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.902605 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.902587 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.902690 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.902597 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 22:49:34.902769 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.902678 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.906695 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.906675 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 22:49:34.907172 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.907156 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:34.908484 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.908466 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 22:49:34.908653 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.908637 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 22:49:34.908859 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.908842 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 22:49:34.910226 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.910211 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:34.910713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.910702 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:34.911310 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911293 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 22:49:34.911462 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911444 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 22:49:34.911531 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911498 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 22:49:34.911531 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911508 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kg7vs\"" Oct 31 22:49:34.911617 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911532 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6jkj5\"" Oct 31 22:49:34.911617 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911538 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 22:49:34.911760 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911742 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 22:49:34.912002 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.911984 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:34.912158 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.912145 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 22:49:34.912737 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.912724 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.913398 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.913378 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.913466 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.913449 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j2f6b\"" Oct 31 22:49:34.913466 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.913459 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.913585 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.913490 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.913966 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.913947 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 22:49:34.914017 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.913985 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 22:49:34.914100 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.914083 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.915312 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915290 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 22:49:34.915312 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915305 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 22:49:34.915610 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915590 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 22:49:34.915702 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915652 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 22:49:34.915702 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915666 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.915702 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915668 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 22:49:34.915702 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915668 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 22:49:34.915847 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915704 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 22:49:34.915847 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915696 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.915847 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.915697 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 22:49:34.916060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.916047 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.916883 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.916869 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.916883 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.916881 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 22:49:34.917000 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.916870 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.917000 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.916944 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.917290 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917276 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.917337 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917330 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.917385 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917355 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6qjt\"" Oct 31 22:49:34.917385 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917371 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.917385 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917362 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 22:49:34.917483 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917372 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.917483 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.917359 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.918516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918504 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fzlj2\"" Oct 31 22:49:34.918871 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918856 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.918904 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918886 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.918962 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918949 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hdfv2\"" Oct 31 22:49:34.919009 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918950 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.919009 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918975 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 22:49:34.919102 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918983 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 22:49:34.919102 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919019 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.919102 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.918993 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m4zzn\"" Oct 31 22:49:34.919102 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919045 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 22:49:34.919263 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919116 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2gb2z\"" Oct 31 22:49:34.919263 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919139 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 22:49:34.919263 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919154 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.919263 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919158 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.919263 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.919170 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkn4d\"" Oct 31 22:49:34.920676 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.920662 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.920725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.920715 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 22:49:34.920756 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.920725 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 22:49:34.920887 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.920876 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.922317 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.922301 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.927536 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.927515 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 22:49:34.927536 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.927526 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 22:49:34.927896 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.927880 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.928082 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.928069 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 22:49:34.929106 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929072 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 22:49:34.929106 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929098 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 22:49:34.929644 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929631 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.929644 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929639 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.929729 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929667 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.929774 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929761 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.929804 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929767 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.929873 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929860 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 22:49:34.929933 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929898 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.929933 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.929909 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 22:49:34.931243 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931231 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.931325 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931259 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.931325 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931298 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 22:49:34.931497 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931482 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 22:49:34.931620 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931541 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 22:49:34.931684 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931644 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 22:49:34.931684 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931675 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 22:49:34.931790 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931700 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9xrd\"" Oct 31 22:49:34.931790 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931699 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ssqvr\"" Oct 31 22:49:34.931790 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931720 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s2g2v\"" Oct 31 22:49:34.931790 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.931737 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbrj5\"" Oct 31 22:49:34.932871 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.932855 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jfvz2\"" Oct 31 22:49:34.933266 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933246 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gr9dz\"" Oct 31 22:49:34.933266 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933257 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-447ws\"" Oct 31 22:49:34.933266 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933251 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pn67g\"" Oct 31 22:49:34.933519 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933276 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2drsv\"" Oct 31 22:49:34.933519 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933297 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gdsw9\"" Oct 31 22:49:34.933519 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933371 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7g25v\"" Oct 31 22:49:34.933519 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933374 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 22:49:34.933519 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.933496 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-stxgn\"" Oct 31 22:49:34.934961 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.934947 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 22:49:34.935049 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.935026 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 22:49:34.936625 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.936602 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 22:49:34.938846 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.938831 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 22:49:34.938846 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.938853 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 22:49:34.939298 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.939284 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 22:49:34.984036 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.984010 5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:34.995691 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.995673 5043 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 22:49:34.996206 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:34.996194 5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 22:49:34.996246 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:34.996229 5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.us-east-2.compute.internal" Oct 31 22:49:35.011518 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011496 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67c4f288-6571-456f-8aa8-d6f5947eb14d-config-volume\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.011632 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011528 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-run\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.011632 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011565 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovn-node-metrics-cert\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.011749 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011675 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01689403-7698-4f38-b4c7-8a3603742ca8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:35.011749 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011715 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfklt\" (UniqueName: \"kubernetes.io/projected/e4043168-4d17-4609-bc2f-37951b7480b4-kube-api-access-kfklt\") pod \"kas-connection-checker-685895c7c9-k5t2f\" (UID: \"e4043168-4d17-4609-bc2f-37951b7480b4\") " pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:35.011820 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011746 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.011820 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011771 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bjc\" (UniqueName: \"kubernetes.io/projected/3ce54792-3b37-4895-a39a-d233a7e3d4ee-kube-api-access-q4bjc\") pod \"kas-connection-checker-685895c7c9-gftjj\" (UID: \"3ce54792-3b37-4895-a39a-d233a7e3d4ee\") " pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:35.011820 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011795 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e8e748-f088-43d1-8011-bf6c8f91ec59-tmp-dir\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.011927 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011819 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a940d2c-fc2c-40ae-a8bd-5916089eb925-konnectivity-ca\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:35.011927 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011864 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-socket-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.011927 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011880 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovn-node-metrics-cert\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.011927 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011890 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.011927 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011900 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01689403-7698-4f38-b4c7-8a3603742ca8-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:35.011927 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011916 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011941 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-key\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011966 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad03e4e-7139-463f-baf7-752544c6763f-serving-cert\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011989 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9779be5d-063f-4f06-8e58-746db277ce66-host-slash\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011994 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e8e748-f088-43d1-8011-bf6c8f91ec59-tmp-dir\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012011 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-modprobe-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012033 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.011915 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67c4f288-6571-456f-8aa8-d6f5947eb14d-config-volume\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012050 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-multus\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.012111 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012102 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-lib-modules\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012137 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dfd0ee8-e42a-4bbd-b355-44262861da53-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012257 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a940d2c-fc2c-40ae-a8bd-5916089eb925-konnectivity-ca\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012265 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012299 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b06c9464-bdf9-493a-8d44-37cca42409d1-config\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012308 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad03e4e-7139-463f-baf7-752544c6763f-serving-cert\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012327 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37b0f64a-6235-4472-b7e2-a2d2a08fd388-host\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012353 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-bin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012368 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012383 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/756d759c-3587-4ff9-9abb-f24c46f64077-crio-socket\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012418 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8dfd0ee8-e42a-4bbd-b355-44262861da53-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012455 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-kubelet-config\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012457 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-key\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:35.012512 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012506 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4lb\" (UniqueName: \"kubernetes.io/projected/67c4f288-6571-456f-8aa8-d6f5947eb14d-kube-api-access-5c4lb\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.013060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012853 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-slash\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.013060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012906 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-config\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.013060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012489 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b06c9464-bdf9-493a-8d44-37cca42409d1-config\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:35.013060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012937 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjd2s\" (UniqueName: \"kubernetes.io/projected/7afda5df-8528-4304-aa1d-156aefde0156-kube-api-access-hjd2s\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.013060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012954 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.013060 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.012994 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-conf\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013069 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-kubelet\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013107 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnkd\" (UniqueName: \"kubernetes.io/projected/13e8e748-f088-43d1-8011-bf6c8f91ec59-kube-api-access-nsnkd\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013152 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013204 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-cert\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013275 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013282 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.013375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013339 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-config\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.013725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013440 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqdw\" (UniqueName: \"kubernetes.io/projected/6c707771-1798-4442-9277-cecbc002c007-kube-api-access-nxqdw\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:35.013725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013494 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.013725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013584 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9868\" (UniqueName: \"kubernetes.io/projected/37b0f64a-6235-4472-b7e2-a2d2a08fd388-kube-api-access-g9868\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.013725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013628 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-sys\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.013725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013666 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-tuned\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.013725 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013709 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.014003 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.013750 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmq4\" (UniqueName: \"kubernetes.io/projected/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9-kube-api-access-2kmq4\") pod \"volume-data-source-validator-58c9f5f46d-q7688\" (UID: \"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:35.014056 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014027 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-tuned\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.014106 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014077 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.014154 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014120 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntq5r\" (UniqueName: \"kubernetes.io/projected/756d759c-3587-4ff9-9abb-f24c46f64077-kube-api-access-ntq5r\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014153 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrjl\" (UniqueName: \"kubernetes.io/projected/90d658ad-d248-4853-8bd8-b64076ec9968-kube-api-access-mwrjl\") pod \"network-check-source-6f78d59f99-cbfsw\" (UID: \"90d658ad-d248-4853-8bd8-b64076ec9968\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014181 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-etc-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014212 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-ovn\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014252 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwpf\" (UniqueName: \"kubernetes.io/projected/8a1ca4ab-6864-49ea-af42-3c31f7823285-kube-api-access-plwpf\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014316 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014348 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqq7\" (UniqueName: \"kubernetes.io/projected/aad03e4e-7139-463f-baf7-752544c6763f-kube-api-access-bvqq7\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014400 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014427 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-systemd\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014462 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4t5z\" (UniqueName: \"kubernetes.io/projected/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-kube-api-access-b4t5z\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014496 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.014595 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014527 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cni-binary-copy\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.015129 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014791 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.015129 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014896 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-daemon-config\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.015129 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014851 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.015129 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.014937 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.015129 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015051 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.015364 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015341 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-var-lib-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.015410 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015384 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-node-log\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.015453 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015404 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-binary-copy\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.015453 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015424 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cni-binary-copy\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.015453 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015435 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-system-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.015616 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015469 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-multus-certs\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.015616 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015583 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9779be5d-063f-4f06-8e58-746db277ce66-iptables-alerter-script\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.015697 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015620 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b06c9464-bdf9-493a-8d44-37cca42409d1-serving-cert\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:35.015697 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015649 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-serving-cert\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.015697 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.015681 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-cert\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:35.016108 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016083 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-serving-cert\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.016185 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016168 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9779be5d-063f-4f06-8e58-746db277ce66-iptables-alerter-script\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.016236 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016186 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.016236 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016213 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-systemd-units\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.016332 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016257 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlrr\" (UniqueName: \"kubernetes.io/projected/01689403-7698-4f38-b4c7-8a3603742ca8-kube-api-access-gdlrr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:35.016332 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016297 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.016423 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016330 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a940d2c-fc2c-40ae-a8bd-5916089eb925-agent-certs\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:35.016423 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016383 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcqf\" (UniqueName: \"kubernetes.io/projected/9779be5d-063f-4f06-8e58-746db277ce66-kube-api-access-vfcqf\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.016423 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016414 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.016589 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016540 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.016635 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016591 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a940d2c-fc2c-40ae-a8bd-5916089eb925-agent-certs\") pod \"konnectivity-agent-jpmrk\" (UID: \"5a940d2c-fc2c-40ae-a8bd-5916089eb925\") " pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:35.016678 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016642 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-daemon-config\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.016678 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016660 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.016766 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016709 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.016766 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016738 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cnibin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.016881 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016797 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.016881 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016827 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.016971 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016898 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-snapshots\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.016971 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016929 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgzm\" (UniqueName: \"kubernetes.io/projected/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-kube-api-access-nvgzm\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.016971 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016961 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-script-lib\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.017107 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.016997 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gll\" (UniqueName: \"kubernetes.io/projected/f3c6a49a-2e5b-460c-8c6a-ca303019575a-kube-api-access-x5gll\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.017107 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017009 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.017197 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-registration-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.017197 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017170 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b06c9464-bdf9-493a-8d44-37cca42409d1-serving-cert\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:35.017289 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017208 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.017289 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017266 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-snapshots\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.017447 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017414 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.017514 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017498 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.017582 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017537 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.017628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017586 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-trusted-ca\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.017628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017612 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-ovnkube-script-lib\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.017764 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017748 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.017825 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017787 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-kubernetes\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.017898 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017821 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.017898 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017854 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.017898 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017883 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.018036 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017916 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5c8\" (UniqueName: \"kubernetes.io/projected/39a8980d-f812-4330-84e7-c406c8afa702-kube-api-access-kc5c8\") pod \"migrator-66df5d496b-tllzm\" (UID: \"39a8980d-f812-4330-84e7-c406c8afa702\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:35.018036 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017951 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-netns\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.018036 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.017982 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-sys-fs\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.018036 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018015 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.018208 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018041 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-dbus\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.018208 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018073 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-log-socket\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.018208 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018104 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01689403-7698-4f38-b4c7-8a3603742ca8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:35.018208 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018133 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13e8e748-f088-43d1-8011-bf6c8f91ec59-hosts-file\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.018208 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018161 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-etc-kubernetes\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.018208 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018191 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018224 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/756d759c-3587-4ff9-9abb-f24c46f64077-data-volume\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018257 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-config\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018287 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-netns\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018319 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-stats-auth\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018362 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvv7\" (UniqueName: \"kubernetes.io/projected/8dfd0ee8-e42a-4bbd-b355-44262861da53-kube-api-access-dcvv7\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018396 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjjq\" (UniqueName: \"kubernetes.io/projected/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e-kube-api-access-4kjjq\") pod \"kas-connection-checker-685895c7c9-khbjk\" (UID: \"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e\") " pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:35.018468 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018433 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37b0f64a-6235-4472-b7e2-a2d2a08fd388-serviceca\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.018795 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018496 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-systemd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.018795 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018525 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczl4\" (UniqueName: \"kubernetes.io/projected/c8f7392b-faf8-43ad-a96b-7e95a95d7792-kube-api-access-dczl4\") pod \"migrator-66df5d496b-ttq5f\" (UID: \"c8f7392b-faf8-43ad-a96b-7e95a95d7792\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:35.018950 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018926 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.019014 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.018996 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/756d759c-3587-4ff9-9abb-f24c46f64077-data-volume\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.019140 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019084 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-netd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.019345 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019328 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-config\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.019410 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019327 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-k8s-cni-cncf-io\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.019410 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019404 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019429 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019488 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysconfig\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019536 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-env-overrides\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019599 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019614 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-stats-auth\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019705 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.019849 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019735 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-conf-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019956 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.019995 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020037 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-cabundle\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020091 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01689403-7698-4f38-b4c7-8a3603742ca8-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020114 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-host\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020149 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-tmp\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020148 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a1ca4ab-6864-49ea-af42-3c31f7823285-env-overrides\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020214 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.020257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020247 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-service-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020281 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9qc\" (UniqueName: \"kubernetes.io/projected/b8edf410-4aa2-42ea-bf3f-016002b74ffa-kube-api-access-8t9qc\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020295 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020314 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eec52f11-3785-48d2-92a1-cbd30a6047e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020320 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-signing-cabundle\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020477 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37b0f64a-6235-4472-b7e2-a2d2a08fd388-serviceca\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020581 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eec52f11-3785-48d2-92a1-cbd30a6047e2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020614 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.020720 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020647 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.021022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020747 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fsgl\" (UniqueName: \"kubernetes.io/projected/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-kube-api-access-7fsgl\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:35.021022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020830 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aad03e4e-7139-463f-baf7-752544c6763f-tmp\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.021022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020864 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-var-lib-kubelet\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.021022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020899 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.021022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.020957 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8jn\" (UniqueName: \"kubernetes.io/projected/b06c9464-bdf9-493a-8d44-37cca42409d1-kube-api-access-6k8jn\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:35.021223 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021014 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-system-cni-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.021223 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021177 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cnibin\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.021223 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021204 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aad03e4e-7139-463f-baf7-752544c6763f-service-ca-bundle\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.021223 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021219 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.021402 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021259 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-tmp\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.021402 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021310 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.021735 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021713 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-kubelet\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.021815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021738 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-tmp\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.021815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021796 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-hostroot\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.021876 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021829 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.021876 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021848 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.021938 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021885 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2lq\" (UniqueName: \"kubernetes.io/projected/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-kube-api-access-dr2lq\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.021938 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021921 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-bin\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.022017 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.021951 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-socket-dir-parent\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.022017 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022008 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.022125 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022107 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwwt\" (UniqueName: \"kubernetes.io/projected/c8c42546-d756-4965-96c3-e3e05dd3d89b-kube-api-access-5nwwt\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.022183 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022159 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjq8\" (UniqueName: \"kubernetes.io/projected/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-kube-api-access-nxjq8\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:35.022284 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022267 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.022389 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022367 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.022430 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022407 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-default-certificate\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.022459 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022441 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-device-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.022487 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022465 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c4f288-6571-456f-8aa8-d6f5947eb14d-tmp-dir\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.022526 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022484 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:35.022526 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022505 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.022641 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022580 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c4f288-6571-456f-8aa8-d6f5947eb14d-tmp-dir\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.022694 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022665 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-default-certificate\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.022815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.022796 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce8955fd-a862-461a-bce9-a5be688358ef-original-pull-secret\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.026677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.026659 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9868\" (UniqueName: \"kubernetes.io/projected/37b0f64a-6235-4472-b7e2-a2d2a08fd388-kube-api-access-g9868\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.026770 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.026711 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqdw\" (UniqueName: \"kubernetes.io/projected/6c707771-1798-4442-9277-cecbc002c007-kube-api-access-nxqdw\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:35.026999 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.026981 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bjc\" (UniqueName: \"kubernetes.io/projected/3ce54792-3b37-4895-a39a-d233a7e3d4ee-kube-api-access-q4bjc\") pod \"kas-connection-checker-685895c7c9-gftjj\" (UID: \"3ce54792-3b37-4895-a39a-d233a7e3d4ee\") " pod="kube-system/kas-connection-checker-685895c7c9-gftjj" Oct 31 22:49:35.027187 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.027164 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqq7\" (UniqueName: \"kubernetes.io/projected/aad03e4e-7139-463f-baf7-752544c6763f-kube-api-access-bvqq7\") pod \"insights-operator-7c4986d449-fj2lt\" (UID: \"aad03e4e-7139-463f-baf7-752544c6763f\") " pod="openshift-insights/insights-operator-7c4986d449-fj2lt" Oct 31 22:49:35.027621 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.027606 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnkd\" (UniqueName: \"kubernetes.io/projected/13e8e748-f088-43d1-8011-bf6c8f91ec59-kube-api-access-nsnkd\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.027676 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.027660 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjd2s\" (UniqueName: \"kubernetes.io/projected/7afda5df-8528-4304-aa1d-156aefde0156-kube-api-access-hjd2s\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.029007 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.028988 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4lb\" (UniqueName: \"kubernetes.io/projected/67c4f288-6571-456f-8aa8-d6f5947eb14d-kube-api-access-5c4lb\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.029452 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.029437 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfklt\" (UniqueName: \"kubernetes.io/projected/e4043168-4d17-4609-bc2f-37951b7480b4-kube-api-access-kfklt\") pod \"kas-connection-checker-685895c7c9-k5t2f\" (UID: \"e4043168-4d17-4609-bc2f-37951b7480b4\") " pod="kube-system/kas-connection-checker-685895c7c9-k5t2f" Oct 31 22:49:35.032095 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.032079 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmq4\" (UniqueName: \"kubernetes.io/projected/2d1c440e-90c0-4411-84a4-a5bfaf91d3e9-kube-api-access-2kmq4\") pod \"volume-data-source-validator-58c9f5f46d-q7688\" (UID: \"2d1c440e-90c0-4411-84a4-a5bfaf91d3e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q7688" Oct 31 22:49:35.032930 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.032904 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlrr\" (UniqueName: \"kubernetes.io/projected/01689403-7698-4f38-b4c7-8a3603742ca8-kube-api-access-gdlrr\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9\" (UID: \"01689403-7698-4f38-b4c7-8a3603742ca8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" Oct 31 22:49:35.033003 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.032987 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrjl\" (UniqueName: \"kubernetes.io/projected/90d658ad-d248-4853-8bd8-b64076ec9968-kube-api-access-mwrjl\") pod \"network-check-source-6f78d59f99-cbfsw\" (UID: \"90d658ad-d248-4853-8bd8-b64076ec9968\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbfsw" Oct 31 22:49:35.033162 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.033147 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gll\" (UniqueName: \"kubernetes.io/projected/f3c6a49a-2e5b-460c-8c6a-ca303019575a-kube-api-access-x5gll\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.037244 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.037228 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntq5r\" (UniqueName: \"kubernetes.io/projected/756d759c-3587-4ff9-9abb-f24c46f64077-kube-api-access-ntq5r\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.039103 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.039084 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcqf\" (UniqueName: \"kubernetes.io/projected/9779be5d-063f-4f06-8e58-746db277ce66-kube-api-access-vfcqf\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.039366 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.039350 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwpf\" (UniqueName: \"kubernetes.io/projected/8a1ca4ab-6864-49ea-af42-3c31f7823285-kube-api-access-plwpf\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.043133 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.043117 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4t5z\" (UniqueName: \"kubernetes.io/projected/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-kube-api-access-b4t5z\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.044633 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.044618 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgzm\" (UniqueName: \"kubernetes.io/projected/12ea1ea8-83a9-4c99-a124-5bf66d1f26b5-kube-api-access-nvgzm\") pod \"console-operator-7cd56b8dbc-vkldg\" (UID: \"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:35.044874 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.044856 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") pod \"periodic-gathering-4xsrk-mwv8t\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:49:35.045400 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.045379 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.045846 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.045827 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjjq\" (UniqueName: \"kubernetes.io/projected/6c14a7d4-f9a1-4e58-8ee5-8455aecf175e-kube-api-access-4kjjq\") pod \"kas-connection-checker-685895c7c9-khbjk\" (UID: \"6c14a7d4-f9a1-4e58-8ee5-8455aecf175e\") " pod="kube-system/kas-connection-checker-685895c7c9-khbjk" Oct 31 22:49:35.047290 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.047270 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.047350 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.047307 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.052963 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.052944 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.053049 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.053005 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczl4\" (UniqueName: \"kubernetes.io/projected/c8f7392b-faf8-43ad-a96b-7e95a95d7792-kube-api-access-dczl4\") pod \"migrator-66df5d496b-ttq5f\" (UID: \"c8f7392b-faf8-43ad-a96b-7e95a95d7792\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ttq5f" Oct 31 22:49:35.053300 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.053285 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5c8\" (UniqueName: \"kubernetes.io/projected/39a8980d-f812-4330-84e7-c406c8afa702-kube-api-access-kc5c8\") pod \"migrator-66df5d496b-tllzm\" (UID: \"39a8980d-f812-4330-84e7-c406c8afa702\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tllzm" Oct 31 22:49:35.053691 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.053676 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8jn\" (UniqueName: \"kubernetes.io/projected/b06c9464-bdf9-493a-8d44-37cca42409d1-kube-api-access-6k8jn\") pod \"service-ca-operator-544d749c8c-cth9l\" (UID: \"b06c9464-bdf9-493a-8d44-37cca42409d1\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" Oct 31 22:49:35.057984 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.057965 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9qc\" (UniqueName: \"kubernetes.io/projected/b8edf410-4aa2-42ea-bf3f-016002b74ffa-kube-api-access-8t9qc\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.058924 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.058908 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjq8\" (UniqueName: \"kubernetes.io/projected/1d95fc77-4bb6-4464-97f2-2bd5500d88e1-kube-api-access-nxjq8\") pod \"service-ca-5fffc97bdf-8g88j\" (UID: \"1d95fc77-4bb6-4464-97f2-2bd5500d88e1\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8g88j" Oct 31 22:49:35.062034 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.062017 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwwt\" (UniqueName: \"kubernetes.io/projected/c8c42546-d756-4965-96c3-e3e05dd3d89b-kube-api-access-5nwwt\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.062092 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.062031 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2lq\" (UniqueName: \"kubernetes.io/projected/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-kube-api-access-dr2lq\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.063198 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.063179 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svcv2\" (UniqueName: \"kubernetes.io/projected/7533bb1d-7252-4b45-b915-969a3f216ba4-kube-api-access-svcv2\") pod \"network-check-target-56mmn\" (UID: \"7533bb1d-7252-4b45-b915-969a3f216ba4\") " pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:35.066219 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.066200 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvv7\" (UniqueName: \"kubernetes.io/projected/8dfd0ee8-e42a-4bbd-b355-44262861da53-kube-api-access-dcvv7\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.077181 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.077160 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fsgl\" (UniqueName: \"kubernetes.io/projected/f9191bfb-6d65-4d74-8948-fcdb9ad57b76-kube-api-access-7fsgl\") pod \"ingress-canary-r2hnp\" (UID: \"f9191bfb-6d65-4d74-8948-fcdb9ad57b76\") " pod="openshift-ingress-canary/ingress-canary-r2hnp" Oct 31 22:49:35.119210 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.119172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:35.123279 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123260 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-etc-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123361 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123292 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-ovn\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123361 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123316 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:35.123361 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123340 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123365 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-systemd\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123365 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-etc-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123381 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-ovn\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123402 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123425 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-systemd\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123425 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123438 5043 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123468 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-var-lib-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123483 5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123494 5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123493 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-node-log\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.123516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123519 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-system-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123526 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-var-lib-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123538 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.623516421 +0000 UTC m=+1.853070827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : secret "metrics-daemon-secret" not found Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123489 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123457 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123590 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.62358005 +0000 UTC m=+1.853134444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123589 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-system-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123602 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-node-log\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123612 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.623601167 +0000 UTC m=+1.853155559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123643 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-multus-certs\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123695 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-multus-certs\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123731 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-systemd-units\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123752 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123769 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123787 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123802 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cnibin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123835 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-registration-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123852 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.623840455 +0000 UTC m=+1.853394886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123860 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-systemd-units\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123881 5043 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123886 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-kubernetes\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123898 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-cnibin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123880 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-cni-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123880 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-registration-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.123945 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert podName:eec52f11-3785-48d2-92a1-cbd30a6047e2 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.623929916 +0000 UTC m=+1.853484325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qv9l7" (UID: "eec52f11-3785-48d2-92a1-cbd30a6047e2") : secret "networking-console-plugin-cert" not found Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123936 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-kubernetes\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123967 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.123999 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-netns\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.124001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124022 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-sys-fs\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124042 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-netns\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124045 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124072 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-dbus\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124087 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-sys-fs\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124096 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-log-socket\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124122 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-log-socket\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124126 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13e8e748-f088-43d1-8011-bf6c8f91ec59-hosts-file\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124129 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-dbus\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124099 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124153 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-etc-kubernetes\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124163 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/13e8e748-f088-43d1-8011-bf6c8f91ec59-hosts-file\") pod \"node-resolver-sk99x\" (UID: \"13e8e748-f088-43d1-8011-bf6c8f91ec59\") " pod="openshift-dns/node-resolver-sk99x" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124163 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-etc-kubernetes\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124178 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124208 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-netns\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124234 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-systemd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124247 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-netns\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124236 5043 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124262 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-netd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.124268 5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.124279 5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124285 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-k8s-cni-cncf-io\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124304 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-netd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124310 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysconfig\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.124317 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.62430816 +0000 UTC m=+1.853862572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124276 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-systemd\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124334 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-run-k8s-cni-cncf-io\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124361 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124369 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysconfig\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124381 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-conf-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124400 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.124407 5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124424 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-host\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124441 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-conf-dir\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.124449 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:35.624437097 +0000 UTC m=+1.853991487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124473 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-host\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124477 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124497 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-var-lib-kubelet\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124515 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-system-cni-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124530 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cnibin\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124577 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-kubelet\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124601 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-hostroot\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124626 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-bin\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124636 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-kubelet\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124647 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-socket-dir-parent\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124601 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-var-lib-kubelet\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124671 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124687 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-hostroot\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124602 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-system-cni-dir\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124705 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-cni-bin\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124696 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124748 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124756 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-multus-socket-dir-parent\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124745 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124780 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124787 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-device-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124820 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-run\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124848 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124859 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-socket-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124870 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-run\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124822 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-device-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124888 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124919 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9779be5d-063f-4f06-8e58-746db277ce66-host-slash\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124918 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-socket-dir\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124939 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/756d759c-3587-4ff9-9abb-f24c46f64077-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124957 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9779be5d-063f-4f06-8e58-746db277ce66-host-slash\") pod \"iptables-alerter-q85ln\" (UID: \"9779be5d-063f-4f06-8e58-746db277ce66\") " pod="openshift-network-operator/iptables-alerter-q85ln" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124959 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ffdc8b2-d568-4e04-a6a8-681de62c2ab5-cnibin\") pod \"multus-additional-cni-plugins-gbvzb\" (UID: \"9ffdc8b2-d568-4e04-a6a8-681de62c2ab5\") " pod="openshift-multus/multus-additional-cni-plugins-gbvzb" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.124977 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-modprobe-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125025 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-multus\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125046 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-lib-modules\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125072 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37b0f64a-6235-4472-b7e2-a2d2a08fd388-host\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125075 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-multus\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125096 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-bin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125119 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/756d759c-3587-4ff9-9abb-f24c46f64077-crio-socket\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125120 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-modprobe-d\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125147 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37b0f64a-6235-4472-b7e2-a2d2a08fd388-host\") pod \"node-ca-58lh2\" (UID: \"37b0f64a-6235-4472-b7e2-a2d2a08fd388\") " pod="openshift-image-registry/node-ca-58lh2" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125138 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-kubelet-config\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125169 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3c6a49a-2e5b-460c-8c6a-ca303019575a-host-var-lib-cni-bin\") pod \"multus-8d5v6\" (UID: \"f3c6a49a-2e5b-460c-8c6a-ca303019575a\") " pod="openshift-multus/multus-8d5v6" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125177 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-lib-modules\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125191 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce8955fd-a862-461a-bce9-a5be688358ef-kubelet-config\") pod \"global-pull-secret-syncer-xrfs4\" (UID: \"ce8955fd-a862-461a-bce9-a5be688358ef\") " pod="kube-system/global-pull-secret-syncer-xrfs4" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125192 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-slash\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125221 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-slash\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125223 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/756d759c-3587-4ff9-9abb-f24c46f64077-crio-socket\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125233 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-conf\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125275 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-kubelet\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125301 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125323 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-sys\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125346 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125360 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-etc-sysctl-conf\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.125321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125388 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-run-openvswitch\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.128376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125416 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8c42546-d756-4965-96c3-e3e05dd3d89b-etc-selinux\") pod \"aws-ebs-csi-driver-node-2tbhj\" (UID: \"c8c42546-d756-4965-96c3-e3e05dd3d89b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 22:49:35.128376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125440 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a1ca4ab-6864-49ea-af42-3c31f7823285-host-kubelet\") pod \"ovnkube-node-f6f9f\" (UID: \"8a1ca4ab-6864-49ea-af42-3c31f7823285\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.128376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.125442 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3-sys\") pod \"tuned-gljgz\" (UID: \"a8d96fb7-909c-4e64-b5e8-8cdeccf7eca3\") " pod="openshift-cluster-node-tuning-operator/tuned-gljgz" Oct 31 22:49:35.128376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.126606 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7afda5df-8528-4304-aa1d-156aefde0156-metrics-certs\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.128376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.127078 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67c4f288-6571-456f-8aa8-d6f5947eb14d-metrics-tls\") pod \"dns-default-xctjg\" (UID: \"67c4f288-6571-456f-8aa8-d6f5947eb14d\") " pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.128376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.127252 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8edf410-4aa2-42ea-bf3f-016002b74ffa-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5blk9\" (UID: \"b8edf410-4aa2-42ea-bf3f-016002b74ffa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.198742 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.198716 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" Oct 31 22:49:35.199826 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.199807 5043 scope.go:122] "RemoveContainer" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" Oct 31 22:49:35.211034 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.211014 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:35.255344 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.255313 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:35.319946 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.319925 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:35.463094 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.463003 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2hnp_f9191bfb-6d65-4d74-8948-fcdb9ad57b76/serve-healthcheck-canary/0.log" Oct 31 22:49:35.492250 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.492223 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9"] Oct 31 22:49:35.527796 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.527769 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xctjg"] Oct 31 22:49:35.528375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.528361 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:49:35.530816 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:35.530794 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c4f288_6571_456f_8aa8_d6f5947eb14d.slice/crio-21f6d978356a0542466fcf955ca61f0ceb6ac72867d3113bb5253ae16cba681e WatchSource:0}: Error finding container 21f6d978356a0542466fcf955ca61f0ceb6ac72867d3113bb5253ae16cba681e: Status 404 returned error can't find the container with id 21f6d978356a0542466fcf955ca61f0ceb6ac72867d3113bb5253ae16cba681e Oct 31 22:49:35.629855 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.629829 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:35.630005 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.629861 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:35.630005 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.629886 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:35.630005 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.629965 5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:35.630005 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.629968 5043 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 22:49:35.630005 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.629991 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:35.630005 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.629993 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:35.630283 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630024 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.630009655 +0000 UTC m=+2.859564058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : secret "metrics-daemon-secret" not found Oct 31 22:49:35.630283 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630052 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.630035029 +0000 UTC m=+2.859589418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:35.630283 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.629975 5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786cddd9dc-dfvvd: secret "image-registry-tls" not found Oct 31 22:49:35.630283 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630095 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls podName:0ec08e29-8387-4709-b902-87580b7cc86a nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.630083403 +0000 UTC m=+2.859637815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls") pod "image-registry-786cddd9dc-dfvvd" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a") : secret "image-registry-tls" not found Oct 31 22:49:35.630283 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630205 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.630194006 +0000 UTC m=+2.859748396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:35.630493 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.630398 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.630493 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.630440 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:35.630493 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.630477 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:35.630660 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630596 5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 22:49:35.630660 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630614 5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c6987c8-t24sb: secret "image-registry-tls" not found Oct 31 22:49:35.630660 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630654 5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:35.630758 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630661 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls podName:1e299d0d-850e-4a45-b03b-24071e0d3459 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.630648074 +0000 UTC m=+2.860202478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls") pod "image-registry-5c6987c8-t24sb" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459") : secret "image-registry-tls" not found Oct 31 22:49:35.630758 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:35.630700 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:36.630691532 +0000 UTC m=+2.860245924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:35.632830 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.632806 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eec52f11-3785-48d2-92a1-cbd30a6047e2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qv9l7\" (UID: \"eec52f11-3785-48d2-92a1-cbd30a6047e2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.663592 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.663568 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tllzm_39a8980d-f812-4330-84e7-c406c8afa702/migrator/0.log" Oct 31 22:49:35.801994 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.801957 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" Oct 31 22:49:35.864326 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.864241 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ttq5f_c8f7392b-faf8-43ad-a96b-7e95a95d7792/migrator/0.log" Oct 31 22:49:35.904172 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.904137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:35.988797 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.988773 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:49:35.988956 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.988857 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" event={"ID":"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5","Type":"ContainerStarted","Data":"9a887c9eb4fd3578596aa13886ac4a95160e1df5231b036f194396c2cbc800d5"} Oct 31 22:49:35.990169 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.990147 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" event={"ID":"b8edf410-4aa2-42ea-bf3f-016002b74ffa","Type":"ContainerStarted","Data":"8219b316cb51548d62306fbdb21ddfc5cf3cb49e2501d77ad8f697fab4704820"} Oct 31 22:49:35.993712 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:35.993682 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xctjg" event={"ID":"67c4f288-6571-456f-8aa8-d6f5947eb14d","Type":"ContainerStarted","Data":"21f6d978356a0542466fcf955ca61f0ceb6ac72867d3113bb5253ae16cba681e"} Oct 31 22:49:36.049035 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.049007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:36.064788 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.064733 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9_01689403-7698-4f38-b4c7-8a3603742ca8/kube-storage-version-migrator-operator/0.log" Oct 31 22:49:36.092359 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.092335 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qv9l7"] Oct 31 22:49:36.095905 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:36.095877 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec52f11_3785_48d2_92a1_cbd30a6047e2.slice/crio-82074905d85160f936b8e1d0033982a9146a76817456e966e15dc4e51c2c4021 WatchSource:0}: Error finding container 82074905d85160f936b8e1d0033982a9146a76817456e966e15dc4e51c2c4021: Status 404 returned error can't find the container with id 82074905d85160f936b8e1d0033982a9146a76817456e966e15dc4e51c2c4021 Oct 31 22:49:36.183916 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.183832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:36.323280 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.323202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:36.461816 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.461780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:36.602534 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.602419 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:36.639790 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.639758 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:36.639963 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.639812 5043 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 22:49:36.639963 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.639869 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:36.639963 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.639876 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs podName:6c707771-1798-4442-9277-cecbc002c007 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:38.639858205 +0000 UTC m=+4.869412612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs") pod "network-metrics-daemon-cz8ch" (UID: "6c707771-1798-4442-9277-cecbc002c007") : secret "metrics-daemon-secret" not found Oct 31 22:49:36.639963 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.639902 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:38.639892682 +0000 UTC m=+4.869447071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:36.639963 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.639811 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:36.640245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.639972 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:36.640245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.640009 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:36.640245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.640058 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:36.640245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.640095 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:36.640916 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.640266 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:38.640252078 +0000 UTC m=+4.869806484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:36.640916 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.640287 5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:36.640916 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:36.640323 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls podName:8dfd0ee8-e42a-4bbd-b355-44262861da53 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:38.64031285 +0000 UTC m=+4.869867244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2l28w" (UID: "8dfd0ee8-e42a-4bbd-b355-44262861da53") : secret "cluster-monitoring-operator-tls" not found Oct 31 22:49:36.643605 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.643577 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"image-registry-5c6987c8-t24sb\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:36.643908 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.643888 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"image-registry-786cddd9dc-dfvvd\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:36.701443 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.701414 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:36.710745 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.710717 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:36.735383 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.735340 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:36.877798 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:36.877669 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:37.000160 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:37.000122 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" event={"ID":"eec52f11-3785-48d2-92a1-cbd30a6047e2","Type":"ContainerStarted","Data":"82074905d85160f936b8e1d0033982a9146a76817456e966e15dc4e51c2c4021"} Oct 31 22:49:37.012203 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:37.012169 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-786cddd9dc-dfvvd"] Oct 31 22:49:37.016001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:37.015980 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c6987c8-t24sb"] Oct 31 22:49:37.016824 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:37.016799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:37.019323 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:37.019295 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec08e29_8387_4709_b902_87580b7cc86a.slice/crio-0a50c9bc4509cb01fc7ca3724f090bf2ba615d61f47dd60bb0ae160184eda781 WatchSource:0}: Error finding container 0a50c9bc4509cb01fc7ca3724f090bf2ba615d61f47dd60bb0ae160184eda781: Status 404 returned error can't find the container with id 0a50c9bc4509cb01fc7ca3724f090bf2ba615d61f47dd60bb0ae160184eda781 Oct 31 22:49:37.021537 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:37.021514 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e299d0d_850e_4a45_b03b_24071e0d3459.slice/crio-00fb6ede9b2b1b7d6a2f07abe5a1d4839cd1f6237dc6136015ae8eb18ac00617 WatchSource:0}: Error finding container 00fb6ede9b2b1b7d6a2f07abe5a1d4839cd1f6237dc6136015ae8eb18ac00617: Status 404 returned error can't find the container with id 00fb6ede9b2b1b7d6a2f07abe5a1d4839cd1f6237dc6136015ae8eb18ac00617 Oct 31 22:49:37.152778 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:37.152690 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:38.005557 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.005519 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" event={"ID":"b8edf410-4aa2-42ea-bf3f-016002b74ffa","Type":"ContainerStarted","Data":"7454fdbc232ac3ff21d362ea534ca960cfd2f07348bc8a444cb60279e97edd68"} Oct 31 22:49:38.005993 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.005602 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5blk9" event={"ID":"b8edf410-4aa2-42ea-bf3f-016002b74ffa","Type":"ContainerStarted","Data":"fab8d350ae44c9e86a20f7eeb953b2e135d35f0434d4535b95821b0e134ba0e4"} Oct 31 22:49:38.007106 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.007084 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" event={"ID":"1e299d0d-850e-4a45-b03b-24071e0d3459","Type":"ContainerStarted","Data":"2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c"} Oct 31 22:49:38.007181 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.007122 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" event={"ID":"1e299d0d-850e-4a45-b03b-24071e0d3459","Type":"ContainerStarted","Data":"00fb6ede9b2b1b7d6a2f07abe5a1d4839cd1f6237dc6136015ae8eb18ac00617"} Oct 31 22:49:38.007181 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.007140 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:38.008765 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.008742 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xctjg" event={"ID":"67c4f288-6571-456f-8aa8-d6f5947eb14d","Type":"ContainerStarted","Data":"160a8a1248acdf8396cb0abca92d393751431e8bccea047577ce3af76639eeb3"} Oct 31 22:49:38.008858 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.008786 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xctjg" event={"ID":"67c4f288-6571-456f-8aa8-d6f5947eb14d","Type":"ContainerStarted","Data":"593dcfd9b455743b9eb555ef95bdf00e77cfecd3467f4f09ecdb4b77409cd7c0"} Oct 31 22:49:38.008858 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.008816 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:38.010661 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.010632 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" event={"ID":"0ec08e29-8387-4709-b902-87580b7cc86a","Type":"ContainerStarted","Data":"b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e"} Oct 31 22:49:38.010761 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.010680 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" event={"ID":"0ec08e29-8387-4709-b902-87580b7cc86a","Type":"ContainerStarted","Data":"0a50c9bc4509cb01fc7ca3724f090bf2ba615d61f47dd60bb0ae160184eda781"} Oct 31 22:49:38.012216 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.012193 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qv9l7" event={"ID":"eec52f11-3785-48d2-92a1-cbd30a6047e2","Type":"ContainerStarted","Data":"ec638908b6b10b20682504eb3cf8caac559e38e14f9e8a70bb071396d40d5e87"} Oct 31 22:49:38.246061 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.246030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:38.251992 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.251967 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:38.252114 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.252086 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:49:38.256920 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.256874 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 22:49:38.448431 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.448405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:38.639680 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.639590 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:38.662707 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.662681 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:38.662852 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.662717 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:38.662852 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.662741 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:38.662852 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.662782 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:38.662852 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:38.662831 5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 22:49:38.663047 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:38.662897 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls podName:756d759c-3587-4ff9-9abb-f24c46f64077 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:42.662884166 +0000 UTC m=+8.892438576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtb2p" (UID: "756d759c-3587-4ff9-9abb-f24c46f64077") : secret "insights-runtime-extractor-tls" not found Oct 31 22:49:38.663047 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:49:38.662938 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle podName:7afda5df-8528-4304-aa1d-156aefde0156 nodeName:}" failed. No retries permitted until 2026-10-31 22:49:42.662921694 +0000 UTC m=+8.892476108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle") pod "router-default-8f9bc98f7-xn5rp" (UID: "7afda5df-8528-4304-aa1d-156aefde0156") : configmap references non-existent config key: service-ca.crt Oct 31 22:49:38.665288 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.665260 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dfd0ee8-e42a-4bbd-b355-44262861da53-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2l28w\" (UID: \"8dfd0ee8-e42a-4bbd-b355-44262861da53\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:38.666812 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.666796 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c707771-1798-4442-9277-cecbc002c007-metrics-certs\") pod \"network-metrics-daemon-cz8ch\" (UID: \"6c707771-1798-4442-9277-cecbc002c007\") " pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:38.710852 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.710825 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:49:38.799718 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.799692 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" Oct 31 22:49:38.807692 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.807671 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cz8ch" Oct 31 22:49:38.842291 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:38.842262 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:39.021402 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.021372 5043 generic.go:356] "Generic (PLEG): container finished" podID="aad03e4e-7139-463f-baf7-752544c6763f" containerID="7033cf2ae1df5b527a28b7b3121dd48d89edc4268291194b2749359f86e03372" exitCode=0 Oct 31 22:49:39.021850 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.021452 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" event={"ID":"aad03e4e-7139-463f-baf7-752544c6763f","Type":"ContainerDied","Data":"7033cf2ae1df5b527a28b7b3121dd48d89edc4268291194b2749359f86e03372"} Oct 31 22:49:39.022385 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.022365 5043 scope.go:122] "RemoveContainer" containerID="7033cf2ae1df5b527a28b7b3121dd48d89edc4268291194b2749359f86e03372" Oct 31 22:49:39.049650 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.049630 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:39.079854 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.079827 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w"] Oct 31 22:49:39.085372 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:39.085345 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfd0ee8_e42a_4bbd_b355_44262861da53.slice/crio-128a62b2e42f5beb74495132e6fbbc539f49e3c8948bf764bcc6f611d3f08919 WatchSource:0}: Error finding container 128a62b2e42f5beb74495132e6fbbc539f49e3c8948bf764bcc6f611d3f08919: Status 404 returned error can't find the container with id 128a62b2e42f5beb74495132e6fbbc539f49e3c8948bf764bcc6f611d3f08919 Oct 31 22:49:39.100465 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.100443 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cz8ch"] Oct 31 22:49:39.104091 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:39.104067 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c707771_1798_4442_9277_cecbc002c007.slice/crio-f2e109e10e90b5a13f680996b0450c45ab263999159194b10f9c09463e286103 WatchSource:0}: Error finding container f2e109e10e90b5a13f680996b0450c45ab263999159194b10f9c09463e286103: Status 404 returned error can't find the container with id f2e109e10e90b5a13f680996b0450c45ab263999159194b10f9c09463e286103 Oct 31 22:49:39.666606 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.666517 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:39.667384 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:39.667360 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:49:40.029769 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.029735 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fj2lt" event={"ID":"aad03e4e-7139-463f-baf7-752544c6763f","Type":"ContainerStarted","Data":"0c8728b295709c3d3266a397bcc94dbc10d4967407b670aa5ffee0b6c6473fbc"} Oct 31 22:49:40.031253 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.031223 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cz8ch" event={"ID":"6c707771-1798-4442-9277-cecbc002c007","Type":"ContainerStarted","Data":"f2e109e10e90b5a13f680996b0450c45ab263999159194b10f9c09463e286103"} Oct 31 22:49:40.032636 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.032601 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" event={"ID":"8dfd0ee8-e42a-4bbd-b355-44262861da53","Type":"ContainerStarted","Data":"128a62b2e42f5beb74495132e6fbbc539f49e3c8948bf764bcc6f611d3f08919"} Oct 31 22:49:40.045473 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.045430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:40.048448 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.048430 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:49:40.232068 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.232036 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:40.367434 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.367405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:40.650381 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.650300 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:40.845456 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:40.845434 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:41.038366 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.038333 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2l28w" event={"ID":"8dfd0ee8-e42a-4bbd-b355-44262861da53","Type":"ContainerStarted","Data":"d07ad98e5232ebfa9d02b2deb0c905218bf57cbdff08e6e82a9ff2cdd822341b"} Oct 31 22:49:41.040147 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.040122 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cz8ch" event={"ID":"6c707771-1798-4442-9277-cecbc002c007","Type":"ContainerStarted","Data":"8a0a078f457683236fdf5736d4a2fd43222d11570c9a99d90b857fb0768603c0"} Oct 31 22:49:41.040241 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.040172 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cz8ch" event={"ID":"6c707771-1798-4442-9277-cecbc002c007","Type":"ContainerStarted","Data":"e58ae6a07747978a397b611659e3be9540c74a43f4897acf60b83ee7ef7b025b"} Oct 31 22:49:41.202910 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.202875 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:41.349829 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.349760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:41.498492 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.498461 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:41.850098 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:41.850064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:42.453038 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.453003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:42.526502 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.526477 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 22:49:42.701575 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.701526 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:42.701706 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.701581 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:42.702128 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.702105 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7afda5df-8528-4304-aa1d-156aefde0156-service-ca-bundle\") pod \"router-default-8f9bc98f7-xn5rp\" (UID: \"7afda5df-8528-4304-aa1d-156aefde0156\") " pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:42.703960 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.703883 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/756d759c-3587-4ff9-9abb-f24c46f64077-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtb2p\" (UID: \"756d759c-3587-4ff9-9abb-f24c46f64077\") " pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:42.712862 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.712844 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtb2p" Oct 31 22:49:42.990292 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.990223 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wtb2p"] Oct 31 22:49:42.993263 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:42.993237 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756d759c_3587_4ff9_9abb_f24c46f64077.slice/crio-38b98c4a8a2cee370e10122df9e5d1b016c573dd48b1144564a34bb744421f8f WatchSource:0}: Error finding container 38b98c4a8a2cee370e10122df9e5d1b016c573dd48b1144564a34bb744421f8f: Status 404 returned error can't find the container with id 38b98c4a8a2cee370e10122df9e5d1b016c573dd48b1144564a34bb744421f8f Oct 31 22:49:42.998699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:42.998678 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:43.050801 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:43.050118 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtb2p" event={"ID":"756d759c-3587-4ff9-9abb-f24c46f64077","Type":"ContainerStarted","Data":"38b98c4a8a2cee370e10122df9e5d1b016c573dd48b1144564a34bb744421f8f"} Oct 31 22:49:43.282119 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:43.282089 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8f9bc98f7-xn5rp"] Oct 31 22:49:43.285121 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:43.285094 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afda5df_8528_4304_aa1d_156aefde0156.slice/crio-00f97e6a1ceb3f37e93221710a8fd3167306a53458625c4d12673bc2ea8b74b1 WatchSource:0}: Error finding container 00f97e6a1ceb3f37e93221710a8fd3167306a53458625c4d12673bc2ea8b74b1: Status 404 returned error can't find the container with id 00f97e6a1ceb3f37e93221710a8fd3167306a53458625c4d12673bc2ea8b74b1 Oct 31 22:49:43.569705 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:43.569633 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" Oct 31 22:49:44.054847 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:44.054813 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" event={"ID":"7afda5df-8528-4304-aa1d-156aefde0156","Type":"ContainerStarted","Data":"5106193ed3e2ad49ca55a53cda62c6625603ac001a10b2f0d38402b7c1ee3049"} Oct 31 22:49:44.054996 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:44.054869 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" event={"ID":"7afda5df-8528-4304-aa1d-156aefde0156","Type":"ContainerStarted","Data":"00f97e6a1ceb3f37e93221710a8fd3167306a53458625c4d12673bc2ea8b74b1"} Oct 31 22:49:44.056494 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:44.056473 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtb2p" event={"ID":"756d759c-3587-4ff9-9abb-f24c46f64077","Type":"ContainerStarted","Data":"2d0efe15243844dd6888d0fc9950755a9aa92519138ef8c267de909f296d758e"} Oct 31 22:49:44.056607 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:44.056518 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtb2p" event={"ID":"756d759c-3587-4ff9-9abb-f24c46f64077","Type":"ContainerStarted","Data":"5bf2e7a33eccea5c85fd20ae812011f5bd6bfe7505eecac0e590332eeb7e40c7"} Oct 31 22:49:44.999113 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:44.999071 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:45.001961 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:45.001938 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:46.068431 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:46.068397 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtb2p" event={"ID":"756d759c-3587-4ff9-9abb-f24c46f64077","Type":"ContainerStarted","Data":"4f00b2da973f396c86ee7298b48ccdad90c98ce1259e220ffcc8ca43432b03c5"} Oct 31 22:49:46.068831 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:46.068503 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:49:47.705999 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:47.705970 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:47.706394 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:47.706124 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 22:49:47.707199 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:47.707177 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 22:49:48.024246 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:48.024218 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xctjg" Oct 31 22:49:49.036001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:49.035969 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:49.172314 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:49.172283 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:49.266929 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:49.266895 5043 patch_prober.go:29] interesting pod/image-registry-786cddd9dc-dfvvd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 22:49:49.267062 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:49.266949 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" podUID="0ec08e29-8387-4709-b902-87580b7cc86a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:49:49.789246 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:49.789208 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c6987c8-t24sb"] Oct 31 22:49:49.947934 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:49.947895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:50.143975 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:50.143908 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:51.305708 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:51.305675 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:51.916362 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:51.916326 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:52.062091 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:52.062056 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:53.459108 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:53.459075 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:53.591871 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:53.591831 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:53.730736 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:53.730648 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:55.436816 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:55.436765 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:55.562963 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:55.562930 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xctjg_67c4f288-6571-456f-8aa8-d6f5947eb14d/dns/0.log" Oct 31 22:49:55.572278 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:55.572235 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:55.757134 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:55.757108 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xctjg_67c4f288-6571-456f-8aa8-d6f5947eb14d/kube-rbac-proxy/0.log" Oct 31 22:49:56.156281 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:56.156212 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sk99x_13e8e748-f088-43d1-8011-bf6c8f91ec59/dns-node-resolver/0.log" Oct 31 22:49:56.250858 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:56.250822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:56.560790 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:56.560764 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c6987c8-t24sb_1e299d0d-850e-4a45-b03b-24071e0d3459/registry/0.log" Oct 31 22:49:56.757305 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:56.757277 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-786cddd9dc-dfvvd_0ec08e29-8387-4709-b902-87580b7cc86a/registry/0.log" Oct 31 22:49:57.356674 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:57.356648 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-58lh2_37b0f64a-6235-4472-b7e2-a2d2a08fd388/node-ca/0.log" Oct 31 22:49:57.760238 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:57.760212 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 22:49:57.956664 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:57.956635 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2hnp_f9191bfb-6d65-4d74-8948-fcdb9ad57b76/serve-healthcheck-canary/0.log" Oct 31 22:49:58.634371 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.634338 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:58.786629 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.786587 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:58.845010 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.844934 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8kxz5"] Oct 31 22:49:58.878818 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.878791 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:58.900882 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.900856 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 22:49:58.901000 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.900973 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p9kg6\"" Oct 31 22:49:58.901000 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.900992 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 22:49:58.924824 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.924798 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 22:49:58.924933 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.924859 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 22:49:58.936062 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:58.936037 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:59.046859 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.046831 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.046988 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.046870 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-sys\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.046988 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.046888 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-metrics-client-ca\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.047058 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.047019 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-wtmp\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.047058 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.047045 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-textfile\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.047164 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.047063 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b785r\" (UniqueName: \"kubernetes.io/projected/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-kube-api-access-b785r\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.047164 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.047078 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-root\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.047164 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.047138 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-tls\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.047256 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.047207 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.076320 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.076293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:59.148557 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148477 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148557 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148514 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-sys\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148557 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148532 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-metrics-client-ca\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148599 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-wtmp\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148618 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-sys\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148626 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-textfile\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148649 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b785r\" (UniqueName: \"kubernetes.io/projected/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-kube-api-access-b785r\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148671 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-root\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148693 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-tls\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148720 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.148815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.148755 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-wtmp\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.149193 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.149152 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-textfile\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.149250 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.149230 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-root\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.149325 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.149297 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-metrics-client-ca\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.149325 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.149315 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.150966 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.150942 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.151558 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.151526 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-node-exporter-tls\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.161717 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.161690 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b785r\" (UniqueName: \"kubernetes.io/projected/9c0cbba8-7d58-4923-80ea-55ab6ce5fe70-kube-api-access-b785r\") pod \"node-exporter-8kxz5\" (UID: \"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70\") " pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.228431 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.228406 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:59.230321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.230307 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kxz5" Oct 31 22:49:59.275606 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:49:59.275582 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0cbba8_7d58_4923_80ea_55ab6ce5fe70.slice/crio-1c2e55eddc5750293ada05a45e4e7731361d8a0d4501ad0fe33eccfcd4043113 WatchSource:0}: Error finding container 1c2e55eddc5750293ada05a45e4e7731361d8a0d4501ad0fe33eccfcd4043113: Status 404 returned error can't find the container with id 1c2e55eddc5750293ada05a45e4e7731361d8a0d4501ad0fe33eccfcd4043113 Oct 31 22:49:59.364033 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.364002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:49:59.796489 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.796459 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:49:59.861516 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:49:59.861474 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:00.038299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:00.038270 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:50:00.051577 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:00.051507 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-56mmn" Oct 31 22:50:00.065385 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:00.065355 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:00.132472 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:00.132437 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kxz5" event={"ID":"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70","Type":"ContainerStarted","Data":"1c2e55eddc5750293ada05a45e4e7731361d8a0d4501ad0fe33eccfcd4043113"} Oct 31 22:50:00.298067 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:00.298031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:00.458279 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:00.458181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:01.138210 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:01.138131 5043 generic.go:356] "Generic (PLEG): container finished" podID="9c0cbba8-7d58-4923-80ea-55ab6ce5fe70" containerID="a04c0a0ced1d9c40b67782a494f5af7f68c25d6765744516153229896d6536a4" exitCode=0 Oct 31 22:50:01.138574 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:01.138220 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kxz5" event={"ID":"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70","Type":"ContainerDied","Data":"a04c0a0ced1d9c40b67782a494f5af7f68c25d6765744516153229896d6536a4"} Oct 31 22:50:01.218339 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:01.218314 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:01.369158 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:01.369121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:01.735206 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:01.735165 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:01.879352 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:01.879311 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:02.025129 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:02.025086 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:02.144409 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:02.144371 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kxz5" event={"ID":"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70","Type":"ContainerStarted","Data":"4dc1a8c4c81c29208bc39e0ca0ef93115f0eeea9e41e65b52622e1cb507bfcf2"} Oct 31 22:50:02.144817 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:02.144433 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kxz5" event={"ID":"9c0cbba8-7d58-4923-80ea-55ab6ce5fe70","Type":"ContainerStarted","Data":"c2a273b915b78af681ee26de435c83bc73fdb560fb844df9fadc241f63307496"} Oct 31 22:50:02.164086 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:02.164062 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:02.168883 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:02.168832 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8kxz5" podStartSLOduration=2.602352332 podStartE2EDuration="4.168819474s" totalImagesPullingTime="1.486718432s" totalInitContainerRuntime="79.74871ms" isStatefulPod=true podCreationTimestamp="2026-10-31 22:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:50:02.166965555 +0000 UTC m=+28.396519966" watchObservedRunningTime="2026-10-31 22:50:02.168819474 +0000 UTC m=+28.398373885" Oct 31 22:50:03.439076 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:03.439037 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:03.576350 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:03.576324 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:03.716842 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:03.716771 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:03.857800 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:03.857768 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:05.367189 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:05.367152 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:06.141063 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:06.141033 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:06.283729 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:06.283698 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:06.923068 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:06.923015 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:07.080884 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:07.080852 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:10.447929 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:10.447876 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:10.457515 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:50:10.457484 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:50:10.584582 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:10.584521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:13.114467 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:13.114431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:13.252864 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:13.252832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:13.553294 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:13.553264 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-786cddd9dc-dfvvd"] Oct 31 22:50:13.694260 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:13.694222 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:13.829441 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:13.829356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:14.823299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:14.823260 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" podUID="1e299d0d-850e-4a45-b03b-24071e0d3459" containerName="registry" containerID="cri-o://2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c" gracePeriod=30 Oct 31 22:50:15.047118 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.047089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:15.059450 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.059428 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:50:15.076641 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076575 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076755 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076657 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076755 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076697 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076755 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076713 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076755 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076733 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076973 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076773 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076973 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076810 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.076973 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.076837 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") pod \"1e299d0d-850e-4a45-b03b-24071e0d3459\" (UID: \"1e299d0d-850e-4a45-b03b-24071e0d3459\") " Oct 31 22:50:15.077298 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.077267 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:15.077378 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.077311 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:15.079525 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.079441 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:15.079525 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.079499 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "kube-api-access-plbl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:15.079695 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.079509 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:15.079695 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.079604 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:15.079695 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.079638 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:15.083439 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.083410 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted" pod "1e299d0d-850e-4a45-b03b-24071e0d3459" (UID: "1e299d0d-850e-4a45-b03b-24071e0d3459"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:15.178310 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178280 5043 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-installation-pull-secrets\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178310 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178307 5043 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e299d0d-850e-4a45-b03b-24071e0d3459-ca-trust-extracted\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178317 5043 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-certificates\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178328 5043 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-bound-sa-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178343 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plbl9\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-kube-api-access-plbl9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178355 5043 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e299d0d-850e-4a45-b03b-24071e0d3459-registry-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178364 5043 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e299d0d-850e-4a45-b03b-24071e0d3459-image-registry-private-configuration\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.178474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.178373 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e299d0d-850e-4a45-b03b-24071e0d3459-trusted-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:15.183177 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.183153 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:15.200693 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.200672 5043 generic.go:356] "Generic (PLEG): container finished" podID="1e299d0d-850e-4a45-b03b-24071e0d3459" containerID="2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c" exitCode=0 Oct 31 22:50:15.200770 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.200707 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" event={"ID":"1e299d0d-850e-4a45-b03b-24071e0d3459","Type":"ContainerDied","Data":"2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c"} Oct 31 22:50:15.200770 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.200727 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" Oct 31 22:50:15.200770 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.200769 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c6987c8-t24sb" event={"ID":"1e299d0d-850e-4a45-b03b-24071e0d3459","Type":"ContainerDied","Data":"00fb6ede9b2b1b7d6a2f07abe5a1d4839cd1f6237dc6136015ae8eb18ac00617"} Oct 31 22:50:15.200863 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.200788 5043 scope.go:122] "RemoveContainer" containerID="2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c" Oct 31 22:50:15.209890 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.209873 5043 scope.go:122] "RemoveContainer" containerID="2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c" Oct 31 22:50:15.210129 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:50:15.210111 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c\": container with ID starting with 2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c not found: ID does not exist" containerID="2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c" Oct 31 22:50:15.210177 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.210138 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c"} err="failed to get container status \"2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c\": rpc error: code = NotFound desc = could not find container \"2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c\": container with ID starting with 2225cb6d180fb41876cd6e34e1a4d6b39a01dd3b929110c8710fd2f42545305c not found: ID does not exist" Oct 31 22:50:15.223574 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.223540 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c6987c8-t24sb"] Oct 31 22:50:15.231880 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.231862 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c6987c8-t24sb"] Oct 31 22:50:15.933127 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:15.933092 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e299d0d-850e-4a45-b03b-24071e0d3459" path="/var/lib/kubelet/pods/1e299d0d-850e-4a45-b03b-24071e0d3459/volumes" Oct 31 22:50:17.231668 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:17.231628 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:17.372349 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:17.372317 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:18.816957 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:18.816926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:18.953701 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:18.953673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:21.911304 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:21.911263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:22.047500 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.047465 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:22.490067 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.490031 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4xsrk-mwv8t"] Oct 31 22:50:22.490300 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.490275 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" podUID="9a36a016-71e1-4a23-b4ca-768e186091e3" containerName="insights-gathering" containerID="cri-o://62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416" gracePeriod=30 Oct 31 22:50:22.746252 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.746227 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:50:22.841067 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.841038 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:22.844088 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844067 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") pod \"9a36a016-71e1-4a23-b4ca-768e186091e3\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " Oct 31 22:50:22.844146 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844135 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") pod \"9a36a016-71e1-4a23-b4ca-768e186091e3\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " Oct 31 22:50:22.844293 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844279 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") pod \"9a36a016-71e1-4a23-b4ca-768e186091e3\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " Oct 31 22:50:22.844356 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844342 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") pod \"9a36a016-71e1-4a23-b4ca-768e186091e3\" (UID: \"9a36a016-71e1-4a23-b4ca-768e186091e3\") " Oct 31 22:50:22.844418 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844398 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path" pod "9a36a016-71e1-4a23-b4ca-768e186091e3" (UID: "9a36a016-71e1-4a23-b4ca-768e186091e3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:22.844476 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844402 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle" pod "9a36a016-71e1-4a23-b4ca-768e186091e3" (UID: "9a36a016-71e1-4a23-b4ca-768e186091e3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:22.844625 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844604 5043 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-service-ca-bundle\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:22.844703 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844629 5043 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9a36a016-71e1-4a23-b4ca-768e186091e3-archives-path\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:22.844703 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.844671 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle" pod "9a36a016-71e1-4a23-b4ca-768e186091e3" (UID: "9a36a016-71e1-4a23-b4ca-768e186091e3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:22.846603 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.846579 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8" pod "9a36a016-71e1-4a23-b4ca-768e186091e3" (UID: "9a36a016-71e1-4a23-b4ca-768e186091e3"). InnerVolumeSpecName "kube-api-access-46qm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:22.944945 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.944918 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a36a016-71e1-4a23-b4ca-768e186091e3-trusted-ca-bundle\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:22.944945 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:22.944941 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qm8\" (UniqueName: \"kubernetes.io/projected/9a36a016-71e1-4a23-b4ca-768e186091e3-kube-api-access-46qm8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:23.004953 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.004880 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:23.235216 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.235179 5043 generic.go:356] "Generic (PLEG): container finished" podID="9a36a016-71e1-4a23-b4ca-768e186091e3" containerID="62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416" exitCode=2 Oct 31 22:50:23.235375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.235219 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" Oct 31 22:50:23.235375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.235261 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" event={"ID":"9a36a016-71e1-4a23-b4ca-768e186091e3","Type":"ContainerDied","Data":"62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416"} Oct 31 22:50:23.235375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.235303 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsrk-mwv8t" event={"ID":"9a36a016-71e1-4a23-b4ca-768e186091e3","Type":"ContainerDied","Data":"a0bb1f3df96c1d1d53598b1166da5932b970640001d2e2edf0ae5b52d62f1417"} Oct 31 22:50:23.235375 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.235322 5043 scope.go:122] "RemoveContainer" containerID="62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416" Oct 31 22:50:23.243841 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.243824 5043 scope.go:122] "RemoveContainer" containerID="62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416" Oct 31 22:50:23.244125 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:50:23.244097 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416\": container with ID starting with 62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416 not found: ID does not exist" containerID="62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416" Oct 31 22:50:23.244191 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.244133 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416"} err="failed to get container status \"62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416\": rpc error: code = NotFound desc = could not find container \"62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416\": container with ID starting with 62cd4edfb92d139aefacccdafb8acc55173ba1b575cb9056769a2b27a64f7416 not found: ID does not exist" Oct 31 22:50:23.259241 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.259185 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4xsrk-mwv8t"] Oct 31 22:50:23.266801 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.263758 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4xsrk-mwv8t"] Oct 31 22:50:23.932786 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:23.932752 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a36a016-71e1-4a23-b4ca-768e186091e3" path="/var/lib/kubelet/pods/9a36a016-71e1-4a23-b4ca-768e186091e3/volumes" Oct 31 22:50:36.913719 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:36.913684 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:37.050762 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:37.050726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:38.576843 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.576777 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" podUID="0ec08e29-8387-4709-b902-87580b7cc86a" containerName="registry" containerID="cri-o://b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e" gracePeriod=30 Oct 31 22:50:38.823242 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.823216 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:50:38.857618 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857535 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857738 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857627 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857738 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857688 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857844 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857733 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857844 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857773 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857844 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857803 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857981 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857860 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.857981 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.857931 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") pod \"0ec08e29-8387-4709-b902-87580b7cc86a\" (UID: \"0ec08e29-8387-4709-b902-87580b7cc86a\") " Oct 31 22:50:38.858881 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.858854 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:38.859006 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.858914 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:50:38.860350 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.860323 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:38.860444 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.860359 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "kube-api-access-wzfqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:38.860444 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.860394 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:38.860626 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.860603 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:50:38.861011 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.860994 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:50:38.867192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.867073 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted" pod "0ec08e29-8387-4709-b902-87580b7cc86a" (UID: "0ec08e29-8387-4709-b902-87580b7cc86a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:50:38.959085 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959057 5043 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-installation-pull-secrets\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959085 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959084 5043 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-bound-sa-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959085 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959094 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfqz\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-kube-api-access-wzfqz\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959104 5043 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0ec08e29-8387-4709-b902-87580b7cc86a-ca-trust-extracted\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959112 5043 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0ec08e29-8387-4709-b902-87580b7cc86a-registry-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959121 5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-trusted-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959130 5043 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0ec08e29-8387-4709-b902-87580b7cc86a-registry-certificates\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:38.959299 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:38.959140 5043 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0ec08e29-8387-4709-b902-87580b7cc86a-image-registry-private-configuration\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:50:39.299791 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.299761 5043 generic.go:356] "Generic (PLEG): container finished" podID="0ec08e29-8387-4709-b902-87580b7cc86a" containerID="b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e" exitCode=0 Oct 31 22:50:39.299958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.299799 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" event={"ID":"0ec08e29-8387-4709-b902-87580b7cc86a","Type":"ContainerDied","Data":"b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e"} Oct 31 22:50:39.299958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.299808 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" Oct 31 22:50:39.299958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.299834 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786cddd9dc-dfvvd" event={"ID":"0ec08e29-8387-4709-b902-87580b7cc86a","Type":"ContainerDied","Data":"0a50c9bc4509cb01fc7ca3724f090bf2ba615d61f47dd60bb0ae160184eda781"} Oct 31 22:50:39.299958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.299853 5043 scope.go:122] "RemoveContainer" containerID="b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e" Oct 31 22:50:39.308843 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.308827 5043 scope.go:122] "RemoveContainer" containerID="b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e" Oct 31 22:50:39.309116 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:50:39.309095 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e\": container with ID starting with b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e not found: ID does not exist" containerID="b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e" Oct 31 22:50:39.309181 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.309124 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e"} err="failed to get container status \"b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e\": rpc error: code = NotFound desc = could not find container \"b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e\": container with ID starting with b579eabf15f5cd2508fcdb84cf8b2a00450411e8a50117f8f3fe47217777bc4e not found: ID does not exist" Oct 31 22:50:39.322470 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.322442 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-786cddd9dc-dfvvd"] Oct 31 22:50:39.326196 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.326176 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-786cddd9dc-dfvvd"] Oct 31 22:50:39.933477 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:39.933441 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec08e29-8387-4709-b902-87580b7cc86a" path="/var/lib/kubelet/pods/0ec08e29-8387-4709-b902-87580b7cc86a/volumes" Oct 31 22:50:40.454795 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:50:40.454755 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:50:43.317721 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:43.317687 5043 generic.go:356] "Generic (PLEG): container finished" podID="b06c9464-bdf9-493a-8d44-37cca42409d1" containerID="f91d48dabb17b85c59b1c4a65404adf3c4841a9ff4f4ae68854f0739e7a009c4" exitCode=0 Oct 31 22:50:43.318090 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:43.317734 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" event={"ID":"b06c9464-bdf9-493a-8d44-37cca42409d1","Type":"ContainerDied","Data":"f91d48dabb17b85c59b1c4a65404adf3c4841a9ff4f4ae68854f0739e7a009c4"} Oct 31 22:50:43.318090 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:43.318051 5043 scope.go:122] "RemoveContainer" containerID="f91d48dabb17b85c59b1c4a65404adf3c4841a9ff4f4ae68854f0739e7a009c4" Oct 31 22:50:43.560138 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:43.560105 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:43.700558 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:43.700520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:44.324202 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:44.324168 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cth9l" event={"ID":"b06c9464-bdf9-493a-8d44-37cca42409d1","Type":"ContainerStarted","Data":"adeb7f8e3fb4b2bf8e92c04825ce04e4db2cb65affe68de4133a044605047394"} Oct 31 22:50:48.343149 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:48.343068 5043 generic.go:356] "Generic (PLEG): container finished" podID="01689403-7698-4f38-b4c7-8a3603742ca8" containerID="0d163879de497c028af2f928ae7f07a01c467c75afa8cca23ce814aca7998f91" exitCode=0 Oct 31 22:50:48.343491 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:48.343145 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" event={"ID":"01689403-7698-4f38-b4c7-8a3603742ca8","Type":"ContainerDied","Data":"0d163879de497c028af2f928ae7f07a01c467c75afa8cca23ce814aca7998f91"} Oct 31 22:50:48.343491 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:48.343454 5043 scope.go:122] "RemoveContainer" containerID="0d163879de497c028af2f928ae7f07a01c467c75afa8cca23ce814aca7998f91" Oct 31 22:50:49.349476 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:49.349438 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" event={"ID":"01689403-7698-4f38-b4c7-8a3603742ca8","Type":"ContainerStarted","Data":"ee041d9111a4e24773377b5d519f30c2647ccd615041739a210e824bb0a68f7c"} Oct 31 22:50:51.641271 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:51.641223 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:51.775087 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:51.775052 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:51.996866 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:51.996832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:50:52.131567 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:50:52.131518 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:01.896669 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:01.896634 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:10.454127 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:51:10.454091 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:51:22.896888 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:22.896849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:24.249537 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:24.249496 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:24.389136 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:24.389108 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:28.822225 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:28.822191 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:28.959404 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:28.959374 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:36.011045 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:36.011010 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:36.148743 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:36.148711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:37.732803 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:37.732764 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:37.867861 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:37.867823 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:39.634691 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:39.634659 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:39.767846 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:39.767812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:40.454386 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:51:40.454342 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:51:41.402452 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:41.402416 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:41.535429 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:41.535399 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:49.074285 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:49.074210 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:51.916928 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:51.916892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:51:52.055213 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:51:52.055178 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:52:10.454941 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:52:10.454899 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:52:19.895458 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:52:19.895421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:52:22.948459 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:52:22.948418 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:52:40.454115 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:52:40.454075 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:53:08.110035 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:08.109999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:08.734728 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:08.734698 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:08.917896 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:08.917862 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:09.064518 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:09.064481 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:10.330205 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:10.330175 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:10.454344 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:53:10.454309 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:53:10.466833 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:10.466809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:11.631721 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:11.631680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:12.514335 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514303 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7"] Oct 31 22:53:12.514619 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514607 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9a36a016-71e1-4a23-b4ca-768e186091e3" containerName="insights-gathering" Oct 31 22:53:12.514677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514620 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a36a016-71e1-4a23-b4ca-768e186091e3" containerName="insights-gathering" Oct 31 22:53:12.514677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514634 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0ec08e29-8387-4709-b902-87580b7cc86a" containerName="registry" Oct 31 22:53:12.514677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514639 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec08e29-8387-4709-b902-87580b7cc86a" containerName="registry" Oct 31 22:53:12.514677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514652 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1e299d0d-850e-4a45-b03b-24071e0d3459" containerName="registry" Oct 31 22:53:12.514677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514657 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e299d0d-850e-4a45-b03b-24071e0d3459" containerName="registry" Oct 31 22:53:12.514831 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514718 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a36a016-71e1-4a23-b4ca-768e186091e3" containerName="insights-gathering" Oct 31 22:53:12.514831 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514726 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec08e29-8387-4709-b902-87580b7cc86a" containerName="registry" Oct 31 22:53:12.514831 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.514735 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e299d0d-850e-4a45-b03b-24071e0d3459" containerName="registry" Oct 31 22:53:12.518006 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.517985 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.522050 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.522028 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 22:53:12.523717 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.523685 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 22:53:12.523820 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.523716 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-h7w87\"" Oct 31 22:53:12.527240 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.527221 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7"] Oct 31 22:53:12.572403 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.572377 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.572494 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.572416 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.572494 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.572442 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5x94\" (UniqueName: \"kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.673351 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.673329 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.673638 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.673365 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.673638 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.673395 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5x94\" (UniqueName: \"kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.673711 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.673678 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.673753 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.673740 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.686305 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.686276 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5x94\" (UniqueName: \"kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:12.834289 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:12.834214 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:13.106450 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.106376 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7"] Oct 31 22:53:13.109415 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:13.109389 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce8aec2_8eed_4095_adfd_0ee79eccec1c.slice/crio-6e4ed00c753134182846cf968e6d9a805cf36aa1d702c924734deccf5a5c7670 WatchSource:0}: Error finding container 6e4ed00c753134182846cf968e6d9a805cf36aa1d702c924734deccf5a5c7670: Status 404 returned error can't find the container with id 6e4ed00c753134182846cf968e6d9a805cf36aa1d702c924734deccf5a5c7670 Oct 31 22:53:13.731376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.731345 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458"] Oct 31 22:53:13.735131 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.735109 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.743454 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.743429 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-tf8bx\"" Oct 31 22:53:13.744713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.744689 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458"] Oct 31 22:53:13.781722 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.781694 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxgh4\" (UniqueName: \"kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.781847 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.781778 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.781847 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.781824 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.882943 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.882916 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.883101 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.882968 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxgh4\" (UniqueName: \"kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.883101 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.883028 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.883360 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.883335 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.883435 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.883372 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.895948 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.895923 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxgh4\" (UniqueName: \"kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:13.906136 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:13.906103 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" event={"ID":"cce8aec2-8eed-4095-adfd-0ee79eccec1c","Type":"ContainerStarted","Data":"6e4ed00c753134182846cf968e6d9a805cf36aa1d702c924734deccf5a5c7670"} Oct 31 22:53:14.056291 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:14.056262 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:14.346350 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:14.346275 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458"] Oct 31 22:53:14.349745 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:14.349716 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd928c662_ecaa_4a04_83af_efdcd5df1df1.slice/crio-f2ccbb60beece1f90e676fd9d4870970405de11848fc4d8231dbae4d7f16df1e WatchSource:0}: Error finding container f2ccbb60beece1f90e676fd9d4870970405de11848fc4d8231dbae4d7f16df1e: Status 404 returned error can't find the container with id f2ccbb60beece1f90e676fd9d4870970405de11848fc4d8231dbae4d7f16df1e Oct 31 22:53:14.912877 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:14.912839 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" event={"ID":"d928c662-ecaa-4a04-83af-efdcd5df1df1","Type":"ContainerStarted","Data":"f2ccbb60beece1f90e676fd9d4870970405de11848fc4d8231dbae4d7f16df1e"} Oct 31 22:53:15.126045 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.126002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:15.276936 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.276896 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:15.555479 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.555394 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z"] Oct 31 22:53:15.559471 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.559448 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.568377 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.568354 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-h6jfg\"" Oct 31 22:53:15.569907 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.569883 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z"] Oct 31 22:53:15.599018 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.598983 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrzd\" (UniqueName: \"kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.599189 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.599045 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.599189 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.599124 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.700411 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.700373 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.700411 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.700419 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.700676 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.700520 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrzd\" (UniqueName: \"kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.701328 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.701304 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.701541 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.701525 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.714462 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.714434 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrzd\" (UniqueName: \"kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.876839 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.876752 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf"] Oct 31 22:53:15.880614 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.880584 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:15.880762 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.880616 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:15.884800 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.884778 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ldmfr\"" Oct 31 22:53:15.894008 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.893943 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf"] Oct 31 22:53:15.904570 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.902612 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:15.904570 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.902684 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pm2\" (UniqueName: \"kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:15.904570 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:15.902744 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.003224 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.003191 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.003639 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.003246 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pm2\" (UniqueName: \"kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.003639 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.003300 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.003766 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.003725 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.004178 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.004154 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.018942 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.018917 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pm2\" (UniqueName: \"kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.185982 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.185905 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z"] Oct 31 22:53:16.190354 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:16.190323 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0a5fbf_9e7c_429e_a671_c162475183f4.slice/crio-437f7368ac861db274b80f9f3de908fdf8fae9ba5c3342da47f2132af6832f24 WatchSource:0}: Error finding container 437f7368ac861db274b80f9f3de908fdf8fae9ba5c3342da47f2132af6832f24: Status 404 returned error can't find the container with id 437f7368ac861db274b80f9f3de908fdf8fae9ba5c3342da47f2132af6832f24 Oct 31 22:53:16.218689 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.218663 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:16.345133 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.343629 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs"] Oct 31 22:53:16.348323 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.347819 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.354200 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.353454 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 22:53:16.361922 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.361864 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs"] Oct 31 22:53:16.362765 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.362732 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nf6c6\"" Oct 31 22:53:16.407207 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407181 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407207 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407210 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407251 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mzdl\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407295 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407319 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407338 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407357 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407428 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.407481 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.407455 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.429883 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.429832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:16.508408 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508372 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508587 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508437 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508587 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508466 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508587 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508491 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508587 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508578 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508842 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508609 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508842 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508646 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508842 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508669 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.508842 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.508714 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mzdl\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.509522 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.509496 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.509664 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.509630 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.510564 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.509877 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.510564 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.510483 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.510564 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.510496 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.512369 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.512338 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.512480 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.512369 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.515136 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.515069 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf"] Oct 31 22:53:16.520621 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:16.520594 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2a0bf3_e89f_4924_8fbb_23484d794bbc.slice/crio-580be4b08debf586608617c6cd6876bf01232845b678775af50794c90fc05907 WatchSource:0}: Error finding container 580be4b08debf586608617c6cd6876bf01232845b678775af50794c90fc05907: Status 404 returned error can't find the container with id 580be4b08debf586608617c6cd6876bf01232845b678775af50794c90fc05907 Oct 31 22:53:16.521985 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.521965 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.522322 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.522264 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mzdl\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl\") pod \"maas-default-gateway-openshift-default-6886b96f9-x9wxs\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.570203 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.570165 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:16.664752 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.664486 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:16.929360 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.929319 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" event={"ID":"4a0a5fbf-9e7c-429e-a671-c162475183f4","Type":"ContainerStarted","Data":"437f7368ac861db274b80f9f3de908fdf8fae9ba5c3342da47f2132af6832f24"} Oct 31 22:53:16.931955 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.931913 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" event={"ID":"3e2a0bf3-e89f-4924-8fbb-23484d794bbc","Type":"ContainerStarted","Data":"580be4b08debf586608617c6cd6876bf01232845b678775af50794c90fc05907"} Oct 31 22:53:16.965098 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:16.965048 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs"] Oct 31 22:53:16.969569 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:16.968746 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dc44da_2acd_4370_a0c7_37b534779995.slice/crio-a50680158a4600e5f5ddc4ba9f70be60a759c7b6964b5970e9e2cba7c9b3ec0c WatchSource:0}: Error finding container a50680158a4600e5f5ddc4ba9f70be60a759c7b6964b5970e9e2cba7c9b3ec0c: Status 404 returned error can't find the container with id a50680158a4600e5f5ddc4ba9f70be60a759c7b6964b5970e9e2cba7c9b3ec0c Oct 31 22:53:17.940868 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:17.940820 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" event={"ID":"11dc44da-2acd-4370-a0c7-37b534779995","Type":"ContainerStarted","Data":"a50680158a4600e5f5ddc4ba9f70be60a759c7b6964b5970e9e2cba7c9b3ec0c"} Oct 31 22:53:19.420279 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:19.420236 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:21.854043 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:21.853994 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:53:21.854458 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:21.854120 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:53:21.854458 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:21.854168 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:53:21.976460 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:21.975665 5043 generic.go:356] "Generic (PLEG): container finished" podID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerID="1ad0fd8f6c69902955ffee5166c160372ef0a6aa659ef0f0e9e27283f850b529" exitCode=0 Oct 31 22:53:21.976460 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:21.975784 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" event={"ID":"cce8aec2-8eed-4095-adfd-0ee79eccec1c","Type":"ContainerDied","Data":"1ad0fd8f6c69902955ffee5166c160372ef0a6aa659ef0f0e9e27283f850b529"} Oct 31 22:53:21.980941 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:21.980327 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" event={"ID":"11dc44da-2acd-4370-a0c7-37b534779995","Type":"ContainerStarted","Data":"c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3"} Oct 31 22:53:22.665535 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.665462 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:22.670236 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.670212 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:22.698366 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.698322 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podStartSLOduration=1.816947244 podStartE2EDuration="6.698310148s" totalImagesPullingTime="4.881362904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:22.103639176 +0000 UTC m=+228.333193587" watchObservedRunningTime="2026-10-31 22:53:22.698310148 +0000 UTC m=+228.927864553" Oct 31 22:53:22.986140 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.986061 5043 generic.go:356] "Generic (PLEG): container finished" podID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerID="1f461a76aa90ab546f46285a7b98c268c0fc0f6e4b7f44c565d4caf61cae68b8" exitCode=0 Oct 31 22:53:22.986606 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.986155 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" event={"ID":"4a0a5fbf-9e7c-429e-a671-c162475183f4","Type":"ContainerDied","Data":"1f461a76aa90ab546f46285a7b98c268c0fc0f6e4b7f44c565d4caf61cae68b8"} Oct 31 22:53:22.988072 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.988029 5043 generic.go:356] "Generic (PLEG): container finished" podID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerID="5870975cfe2236af1ff95441374339c76944c920fd7521595705fd9d3f53394f" exitCode=0 Oct 31 22:53:22.988132 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.988099 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" event={"ID":"d928c662-ecaa-4a04-83af-efdcd5df1df1","Type":"ContainerDied","Data":"5870975cfe2236af1ff95441374339c76944c920fd7521595705fd9d3f53394f"} Oct 31 22:53:22.989811 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.989735 5043 generic.go:356] "Generic (PLEG): container finished" podID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerID="8a67bcf430253749cfc6a9713ca0c85e1eea4927a5d09356cf54fd67c11919f6" exitCode=0 Oct 31 22:53:22.989811 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.989760 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" event={"ID":"3e2a0bf3-e89f-4924-8fbb-23484d794bbc","Type":"ContainerDied","Data":"8a67bcf430253749cfc6a9713ca0c85e1eea4927a5d09356cf54fd67c11919f6"} Oct 31 22:53:22.990669 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.990646 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:22.991836 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:22.991813 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 22:53:23.020531 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:23.020502 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:23.997166 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:23.997128 5043 generic.go:356] "Generic (PLEG): container finished" podID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerID="37a437998e6e440f95918b84e7abca3f19fa02bc69f54cebe24df2eb7865605b" exitCode=0 Oct 31 22:53:23.997658 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:23.997206 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" event={"ID":"3e2a0bf3-e89f-4924-8fbb-23484d794bbc","Type":"ContainerDied","Data":"37a437998e6e440f95918b84e7abca3f19fa02bc69f54cebe24df2eb7865605b"} Oct 31 22:53:25.005892 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:25.005858 5043 generic.go:356] "Generic (PLEG): container finished" podID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerID="456cf357a1d6e4fcb78ea726285d7d5c92ff42c224dd92140793d553a7895f00" exitCode=0 Oct 31 22:53:25.006336 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:25.006190 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" event={"ID":"d928c662-ecaa-4a04-83af-efdcd5df1df1","Type":"ContainerDied","Data":"456cf357a1d6e4fcb78ea726285d7d5c92ff42c224dd92140793d553a7895f00"} Oct 31 22:53:25.009169 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:25.009146 5043 generic.go:356] "Generic (PLEG): container finished" podID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerID="9ae8f7407fb3b7b9413624642ef340a3ae4147eda703cca9c0dc47d2ff657157" exitCode=0 Oct 31 22:53:25.009264 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:25.009220 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" event={"ID":"cce8aec2-8eed-4095-adfd-0ee79eccec1c","Type":"ContainerDied","Data":"9ae8f7407fb3b7b9413624642ef340a3ae4147eda703cca9c0dc47d2ff657157"} Oct 31 22:53:26.019746 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:26.019701 5043 generic.go:356] "Generic (PLEG): container finished" podID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerID="eb53b0619d57b680ec176cd920ae6d71153e7508926c78c61078ec11dcdeb159" exitCode=0 Oct 31 22:53:26.020203 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:26.019806 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" event={"ID":"4a0a5fbf-9e7c-429e-a671-c162475183f4","Type":"ContainerDied","Data":"eb53b0619d57b680ec176cd920ae6d71153e7508926c78c61078ec11dcdeb159"} Oct 31 22:53:33.067820 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:33.067777 5043 generic.go:356] "Generic (PLEG): container finished" podID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerID="24f623d23191d554d446183516c9d29cfffcb4ce72f1502cb38ade71fdd9be39" exitCode=0 Oct 31 22:53:33.068343 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:33.067870 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" event={"ID":"3e2a0bf3-e89f-4924-8fbb-23484d794bbc","Type":"ContainerDied","Data":"24f623d23191d554d446183516c9d29cfffcb4ce72f1502cb38ade71fdd9be39"} Oct 31 22:53:34.222420 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.222392 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:34.282196 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.282163 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util\") pod \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " Oct 31 22:53:34.282378 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.282269 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2\" (UniqueName: \"kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2\") pod \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " Oct 31 22:53:34.282378 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.282351 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle\") pod \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\" (UID: \"3e2a0bf3-e89f-4924-8fbb-23484d794bbc\") " Oct 31 22:53:34.282985 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.282945 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle" pod "3e2a0bf3-e89f-4924-8fbb-23484d794bbc" (UID: "3e2a0bf3-e89f-4924-8fbb-23484d794bbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:34.286816 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.286783 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2" pod "3e2a0bf3-e89f-4924-8fbb-23484d794bbc" (UID: "3e2a0bf3-e89f-4924-8fbb-23484d794bbc"). InnerVolumeSpecName "kube-api-access-z8pm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:34.288288 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.288249 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util" pod "3e2a0bf3-e89f-4924-8fbb-23484d794bbc" (UID: "3e2a0bf3-e89f-4924-8fbb-23484d794bbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:34.383918 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.383886 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8pm2\" (UniqueName: \"kubernetes.io/projected/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-kube-api-access-z8pm2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.384013 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.383924 5043 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-bundle\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:34.384013 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:34.383938 5043 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e2a0bf3-e89f-4924-8fbb-23484d794bbc-util\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:35.079398 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.079315 5043 generic.go:356] "Generic (PLEG): container finished" podID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerID="2980a31f756045b819616244ca707a12aede9bde55770a80e0b93301fad48d42" exitCode=0 Oct 31 22:53:35.079398 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.079378 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" event={"ID":"cce8aec2-8eed-4095-adfd-0ee79eccec1c","Type":"ContainerDied","Data":"2980a31f756045b819616244ca707a12aede9bde55770a80e0b93301fad48d42"} Oct 31 22:53:35.081453 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.081428 5043 generic.go:356] "Generic (PLEG): container finished" podID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerID="7c30332c63b44fba7ff4e5c8b29316918899ddf5d68f292fd63ba62efc637c5d" exitCode=0 Oct 31 22:53:35.081585 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.081514 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" event={"ID":"4a0a5fbf-9e7c-429e-a671-c162475183f4","Type":"ContainerDied","Data":"7c30332c63b44fba7ff4e5c8b29316918899ddf5d68f292fd63ba62efc637c5d"} Oct 31 22:53:35.083574 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.083535 5043 generic.go:356] "Generic (PLEG): container finished" podID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerID="b1e4d804056f6cb41fc215616eaefd05dad510ecdebde64f9d80504a5250e92b" exitCode=0 Oct 31 22:53:35.083677 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.083581 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" event={"ID":"d928c662-ecaa-4a04-83af-efdcd5df1df1","Type":"ContainerDied","Data":"b1e4d804056f6cb41fc215616eaefd05dad510ecdebde64f9d80504a5250e92b"} Oct 31 22:53:35.085323 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.085299 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" event={"ID":"3e2a0bf3-e89f-4924-8fbb-23484d794bbc","Type":"ContainerDied","Data":"580be4b08debf586608617c6cd6876bf01232845b678775af50794c90fc05907"} Oct 31 22:53:35.085398 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.085341 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="580be4b08debf586608617c6cd6876bf01232845b678775af50794c90fc05907" Oct 31 22:53:35.085398 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:35.085345 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71984wf" Oct 31 22:53:36.234205 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.234175 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:36.257351 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.257093 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:36.261869 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.261780 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:36.299745 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.299718 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util\") pod \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " Oct 31 22:53:36.299970 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.299800 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle\") pod \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " Oct 31 22:53:36.299970 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.299846 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94\" (UniqueName: \"kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94\") pod \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\" (UID: \"cce8aec2-8eed-4095-adfd-0ee79eccec1c\") " Oct 31 22:53:36.301273 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.300297 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle" pod "cce8aec2-8eed-4095-adfd-0ee79eccec1c" (UID: "cce8aec2-8eed-4095-adfd-0ee79eccec1c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:36.302290 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.302266 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94" pod "cce8aec2-8eed-4095-adfd-0ee79eccec1c" (UID: "cce8aec2-8eed-4095-adfd-0ee79eccec1c"). InnerVolumeSpecName "kube-api-access-w5x94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:36.306199 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.306172 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util" pod "cce8aec2-8eed-4095-adfd-0ee79eccec1c" (UID: "cce8aec2-8eed-4095-adfd-0ee79eccec1c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:36.401147 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401092 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle\") pod \"d928c662-ecaa-4a04-83af-efdcd5df1df1\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " Oct 31 22:53:36.401147 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401129 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle\") pod \"4a0a5fbf-9e7c-429e-a671-c162475183f4\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " Oct 31 22:53:36.401271 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401181 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util\") pod \"4a0a5fbf-9e7c-429e-a671-c162475183f4\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " Oct 31 22:53:36.401271 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401204 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd\" (UniqueName: \"kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd\") pod \"4a0a5fbf-9e7c-429e-a671-c162475183f4\" (UID: \"4a0a5fbf-9e7c-429e-a671-c162475183f4\") " Oct 31 22:53:36.401271 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401226 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4\" (UniqueName: \"kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4\") pod \"d928c662-ecaa-4a04-83af-efdcd5df1df1\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " Oct 31 22:53:36.401271 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401251 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util\") pod \"d928c662-ecaa-4a04-83af-efdcd5df1df1\" (UID: \"d928c662-ecaa-4a04-83af-efdcd5df1df1\") " Oct 31 22:53:36.401475 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401453 5043 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-util\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.401541 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401478 5043 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cce8aec2-8eed-4095-adfd-0ee79eccec1c-bundle\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.401541 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401492 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5x94\" (UniqueName: \"kubernetes.io/projected/cce8aec2-8eed-4095-adfd-0ee79eccec1c-kube-api-access-w5x94\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.401698 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.401676 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle" pod "d928c662-ecaa-4a04-83af-efdcd5df1df1" (UID: "d928c662-ecaa-4a04-83af-efdcd5df1df1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:36.402066 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.402042 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle" pod "4a0a5fbf-9e7c-429e-a671-c162475183f4" (UID: "4a0a5fbf-9e7c-429e-a671-c162475183f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:36.403769 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.403744 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd" pod "4a0a5fbf-9e7c-429e-a671-c162475183f4" (UID: "4a0a5fbf-9e7c-429e-a671-c162475183f4"). InnerVolumeSpecName "kube-api-access-6jrzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:36.403852 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.403775 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4" pod "d928c662-ecaa-4a04-83af-efdcd5df1df1" (UID: "d928c662-ecaa-4a04-83af-efdcd5df1df1"). InnerVolumeSpecName "kube-api-access-cxgh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:53:36.405474 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.405454 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util" pod "4a0a5fbf-9e7c-429e-a671-c162475183f4" (UID: "4a0a5fbf-9e7c-429e-a671-c162475183f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:36.406837 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.406812 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util" pod "d928c662-ecaa-4a04-83af-efdcd5df1df1" (UID: "d928c662-ecaa-4a04-83af-efdcd5df1df1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:53:36.502775 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.502746 5043 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-util\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.502775 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.502773 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jrzd\" (UniqueName: \"kubernetes.io/projected/4a0a5fbf-9e7c-429e-a671-c162475183f4-kube-api-access-6jrzd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.502894 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.502785 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxgh4\" (UniqueName: \"kubernetes.io/projected/d928c662-ecaa-4a04-83af-efdcd5df1df1-kube-api-access-cxgh4\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.502894 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.502793 5043 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-util\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.502894 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.502802 5043 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d928c662-ecaa-4a04-83af-efdcd5df1df1-bundle\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:36.502894 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:36.502812 5043 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a0a5fbf-9e7c-429e-a671-c162475183f4-bundle\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 22:53:37.095648 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.095621 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" Oct 31 22:53:37.095840 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.095643 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644wk5x7" event={"ID":"cce8aec2-8eed-4095-adfd-0ee79eccec1c","Type":"ContainerDied","Data":"6e4ed00c753134182846cf968e6d9a805cf36aa1d702c924734deccf5a5c7670"} Oct 31 22:53:37.095840 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.095695 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e4ed00c753134182846cf968e6d9a805cf36aa1d702c924734deccf5a5c7670" Oct 31 22:53:37.097713 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.097677 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" event={"ID":"4a0a5fbf-9e7c-429e-a671-c162475183f4","Type":"ContainerDied","Data":"437f7368ac861db274b80f9f3de908fdf8fae9ba5c3342da47f2132af6832f24"} Oct 31 22:53:37.097838 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.097728 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="437f7368ac861db274b80f9f3de908fdf8fae9ba5c3342da47f2132af6832f24" Oct 31 22:53:37.097838 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.097684 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctqq6z" Oct 31 22:53:37.099591 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.099571 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" event={"ID":"d928c662-ecaa-4a04-83af-efdcd5df1df1","Type":"ContainerDied","Data":"f2ccbb60beece1f90e676fd9d4870970405de11848fc4d8231dbae4d7f16df1e"} Oct 31 22:53:37.099689 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.099602 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hh458" Oct 31 22:53:37.099689 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.099605 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ccbb60beece1f90e676fd9d4870970405de11848fc4d8231dbae4d7f16df1e" Oct 31 22:53:37.740266 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.740228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:37.886012 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:37.885980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:38.098908 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.098876 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:38.414935 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.414857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:38.511963 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.511923 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld"] Oct 31 22:53:38.512284 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512270 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="util" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512285 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="util" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512297 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="util" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512302 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="util" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512319 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="util" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512324 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="util" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512331 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="extract" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512335 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="extract" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512341 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="extract" Oct 31 22:53:38.512342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512345 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512352 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512356 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512367 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512371 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512380 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512384 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512389 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512393 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512401 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="util" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512405 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="util" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512411 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512415 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512421 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512425 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="pull" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512484 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d928c662-ecaa-4a04-83af-efdcd5df1df1" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512491 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0a5fbf-9e7c-429e-a671-c162475183f4" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512498 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e2a0bf3-e89f-4924-8fbb-23484d794bbc" containerName="extract" Oct 31 22:53:38.512628 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.512503 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce8aec2-8eed-4095-adfd-0ee79eccec1c" containerName="extract" Oct 31 22:53:38.516929 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.516913 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.526072 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.526041 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 22:53:38.526072 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.526064 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 22:53:38.527533 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.527501 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 22:53:38.527682 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.527631 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 22:53:38.527682 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.527662 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 22:53:38.527798 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.527746 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hg7q6\"" Oct 31 22:53:38.552031 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.552009 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld"] Oct 31 22:53:38.581594 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.581570 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:38.621413 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.621388 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.621511 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.621421 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txt6\" (UniqueName: \"kubernetes.io/projected/0063f042-8457-40ab-97d7-94d1420af56b-kube-api-access-4txt6\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.621511 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.621452 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.621599 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.621581 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0063f042-8457-40ab-97d7-94d1420af56b-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.621639 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.621626 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0063f042-8457-40ab-97d7-94d1420af56b-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.621715 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.621702 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.722565 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.722484 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0063f042-8457-40ab-97d7-94d1420af56b-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.722565 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.722522 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.722706 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.722577 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.722706 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.722609 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txt6\" (UniqueName: \"kubernetes.io/projected/0063f042-8457-40ab-97d7-94d1420af56b-kube-api-access-4txt6\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.722706 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.722642 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.722706 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.722695 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0063f042-8457-40ab-97d7-94d1420af56b-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.723040 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.723021 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0063f042-8457-40ab-97d7-94d1420af56b-manifests\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.724952 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.724926 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0063f042-8457-40ab-97d7-94d1420af56b-tmp\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.725072 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.724979 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.725143 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.725108 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.725232 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.725211 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0063f042-8457-40ab-97d7-94d1420af56b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.727721 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.727699 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:38.735141 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.735120 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txt6\" (UniqueName: \"kubernetes.io/projected/0063f042-8457-40ab-97d7-94d1420af56b-kube-api-access-4txt6\") pod \"opendatahub-operator-controller-manager-7c5c6d88b7-ftgld\" (UID: \"0063f042-8457-40ab-97d7-94d1420af56b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.831091 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.831065 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:38.870329 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:38.870300 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:39.008190 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:39.008162 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:39.136745 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:39.136719 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld"] Oct 31 22:53:39.141636 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:39.141601 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0063f042_8457_40ab_97d7_94d1420af56b.slice/crio-81b8a29078f63b93b36757ecef558834a5d9ae61f8b271b33b00396d3378dbc7 WatchSource:0}: Error finding container 81b8a29078f63b93b36757ecef558834a5d9ae61f8b271b33b00396d3378dbc7: Status 404 returned error can't find the container with id 81b8a29078f63b93b36757ecef558834a5d9ae61f8b271b33b00396d3378dbc7 Oct 31 22:53:40.112396 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:40.112309 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" event={"ID":"0063f042-8457-40ab-97d7-94d1420af56b","Type":"ContainerStarted","Data":"81b8a29078f63b93b36757ecef558834a5d9ae61f8b271b33b00396d3378dbc7"} Oct 31 22:53:40.462541 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:53:40.462411 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:53:41.502440 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:41.502401 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:41.645068 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:41.645033 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:41.819407 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:41.819322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:43.127406 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:43.127374 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" event={"ID":"0063f042-8457-40ab-97d7-94d1420af56b","Type":"ContainerStarted","Data":"981ca249b39708197dab4f25f32a27fa9b0dcd34ac21cb715280c53f661953b5"} Oct 31 22:53:44.133295 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:44.133205 5043 generic.go:356] "Generic (PLEG): container finished" podID="0063f042-8457-40ab-97d7-94d1420af56b" containerID="981ca249b39708197dab4f25f32a27fa9b0dcd34ac21cb715280c53f661953b5" exitCode=0 Oct 31 22:53:44.133744 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:44.133291 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" event={"ID":"0063f042-8457-40ab-97d7-94d1420af56b","Type":"ContainerDied","Data":"981ca249b39708197dab4f25f32a27fa9b0dcd34ac21cb715280c53f661953b5"} Oct 31 22:53:44.133744 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:44.133344 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" event={"ID":"0063f042-8457-40ab-97d7-94d1420af56b","Type":"ContainerStarted","Data":"0cd971070a0d1c8f922c607fc23f0dcfe7f7d5926e5ac627e4fedb9508f43045"} Oct 31 22:53:44.133744 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:44.133401 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:44.180454 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:44.180412 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podStartSLOduration=1.70016055 podStartE2EDuration="6.180400275s" totalImagesPullingTime="4.006120507s" totalInitContainerRuntime="474.119218ms" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:44.177651827 +0000 UTC m=+250.407206237" watchObservedRunningTime="2026-10-31 22:53:44.180400275 +0000 UTC m=+250.409954686" Oct 31 22:53:47.156467 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:47.156439 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:47.295157 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:47.295126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:51.277042 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:51.277003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:53.267774 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:53.267740 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:54.815327 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.815289 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd"] Oct 31 22:53:54.818558 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.818532 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:53:54.824644 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.824620 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jq9db\"" Oct 31 22:53:54.827993 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.827970 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 22:53:54.829392 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.829370 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 22:53:54.829565 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.829529 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 22:53:54.831130 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.831109 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd"] Oct 31 22:53:54.953894 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:54.953864 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwj9n\" (UniqueName: \"kubernetes.io/projected/b4fd03ee-af94-4df1-b834-eced676d664e-kube-api-access-hwj9n\") pod \"dns-operator-controller-manager-8557cbdb84-ss2sd\" (UID: \"b4fd03ee-af94-4df1-b834-eced676d664e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:53:55.054880 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:55.054845 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwj9n\" (UniqueName: \"kubernetes.io/projected/b4fd03ee-af94-4df1-b834-eced676d664e-kube-api-access-hwj9n\") pod \"dns-operator-controller-manager-8557cbdb84-ss2sd\" (UID: \"b4fd03ee-af94-4df1-b834-eced676d664e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:53:55.076213 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:55.076155 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwj9n\" (UniqueName: \"kubernetes.io/projected/b4fd03ee-af94-4df1-b834-eced676d664e-kube-api-access-hwj9n\") pod \"dns-operator-controller-manager-8557cbdb84-ss2sd\" (UID: \"b4fd03ee-af94-4df1-b834-eced676d664e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:53:55.130956 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:55.130928 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:53:55.140926 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:55.140900 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 22:53:55.434315 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:55.434281 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd"] Oct 31 22:53:55.437211 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:55.437184 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fd03ee_af94_4df1_b834_eced676d664e.slice/crio-adf6dcdb814363d658adc759292e2433d75cccdc7efdef753360d0d19416eadd WatchSource:0}: Error finding container adf6dcdb814363d658adc759292e2433d75cccdc7efdef753360d0d19416eadd: Status 404 returned error can't find the container with id adf6dcdb814363d658adc759292e2433d75cccdc7efdef753360d0d19416eadd Oct 31 22:53:56.096988 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:56.096956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:56.183035 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:56.183003 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" event={"ID":"b4fd03ee-af94-4df1-b834-eced676d664e","Type":"ContainerStarted","Data":"adf6dcdb814363d658adc759292e2433d75cccdc7efdef753360d0d19416eadd"} Oct 31 22:53:57.408116 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.408075 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn"] Oct 31 22:53:57.414986 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.414963 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.423266 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.423194 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn"] Oct 31 22:53:57.424074 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.423693 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:53:57.424074 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.423752 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fsp6x\"" Oct 31 22:53:57.424074 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.423830 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 22:53:57.579845 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.579804 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fz5c\" (UniqueName: \"kubernetes.io/projected/0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9-kube-api-access-8fz5c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn\" (UID: \"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.580032 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.580002 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn\" (UID: \"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.681496 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.681419 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn\" (UID: \"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.681496 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.681484 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fz5c\" (UniqueName: \"kubernetes.io/projected/0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9-kube-api-access-8fz5c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn\" (UID: \"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.681827 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.681808 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn\" (UID: \"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.699002 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.698969 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fz5c\" (UniqueName: \"kubernetes.io/projected/0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9-kube-api-access-8fz5c\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn\" (UID: \"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:57.731132 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:57.731102 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" Oct 31 22:53:58.034413 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.034379 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn"] Oct 31 22:53:58.039051 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:58.039023 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdcda9c_bcff_4ded_b4e4_d3291eca8fb9.slice/crio-6270b74cfeeb7c887a105770f7f0fdcff7b7dd418c5fc465018cd89777c5db59 WatchSource:0}: Error finding container 6270b74cfeeb7c887a105770f7f0fdcff7b7dd418c5fc465018cd89777c5db59: Status 404 returned error can't find the container with id 6270b74cfeeb7c887a105770f7f0fdcff7b7dd418c5fc465018cd89777c5db59 Oct 31 22:53:58.195165 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.195129 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" event={"ID":"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9","Type":"ContainerStarted","Data":"6270b74cfeeb7c887a105770f7f0fdcff7b7dd418c5fc465018cd89777c5db59"} Oct 31 22:53:58.317052 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.316979 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:53:58.678744 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.678659 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2"] Oct 31 22:53:58.682134 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.682110 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:53:58.687040 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.687018 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-swhmq\"" Oct 31 22:53:58.711577 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.711542 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2"] Oct 31 22:53:58.789325 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.789303 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdpbl\" (UniqueName: \"kubernetes.io/projected/485a8e4b-a74c-4463-affc-44a6374fb81c-kube-api-access-jdpbl\") pod \"limitador-operator-controller-manager-5778bb4449-ktbm2\" (UID: \"485a8e4b-a74c-4463-affc-44a6374fb81c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:53:58.889912 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.889890 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdpbl\" (UniqueName: \"kubernetes.io/projected/485a8e4b-a74c-4463-affc-44a6374fb81c-kube-api-access-jdpbl\") pod \"limitador-operator-controller-manager-5778bb4449-ktbm2\" (UID: \"485a8e4b-a74c-4463-affc-44a6374fb81c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:53:58.902699 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.902678 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdpbl\" (UniqueName: \"kubernetes.io/projected/485a8e4b-a74c-4463-affc-44a6374fb81c-kube-api-access-jdpbl\") pod \"limitador-operator-controller-manager-5778bb4449-ktbm2\" (UID: \"485a8e4b-a74c-4463-affc-44a6374fb81c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:53:58.994320 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:58.994296 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:53:59.203895 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:59.203851 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" event={"ID":"b4fd03ee-af94-4df1-b834-eced676d664e","Type":"ContainerStarted","Data":"6034260101de910aa610eb5cd1a35ee8b59069b7b6b37f14544aaf6115255d9a"} Oct 31 22:53:59.204052 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:59.204018 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:53:59.226876 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:59.226822 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podStartSLOduration=2.318150486 podStartE2EDuration="5.226805686s" totalImagesPullingTime="2.9086552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:53:59.226211366 +0000 UTC m=+265.455765778" watchObservedRunningTime="2026-10-31 22:53:59.226805686 +0000 UTC m=+265.456360101" Oct 31 22:53:59.268572 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:59.268487 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2"] Oct 31 22:53:59.272185 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:53:59.272159 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485a8e4b_a74c_4463_affc_44a6374fb81c.slice/crio-d193913cd1e0b966c511851324f45ce46ccf50d559291d113d047d36719b6d4e WatchSource:0}: Error finding container d193913cd1e0b966c511851324f45ce46ccf50d559291d113d047d36719b6d4e: Status 404 returned error can't find the container with id d193913cd1e0b966c511851324f45ce46ccf50d559291d113d047d36719b6d4e Oct 31 22:53:59.476462 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:53:59.476385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:00.209881 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:00.209840 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" event={"ID":"0bdcda9c-bcff-4ded-b4e4-d3291eca8fb9","Type":"ContainerStarted","Data":"e70d9def5de54938c61c337d9854a54f730f3e6fc0a14c0deb20477d5af15865"} Oct 31 22:54:00.211254 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:00.211220 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" event={"ID":"485a8e4b-a74c-4463-affc-44a6374fb81c","Type":"ContainerStarted","Data":"d193913cd1e0b966c511851324f45ce46ccf50d559291d113d047d36719b6d4e"} Oct 31 22:54:00.236652 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:00.236608 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-zt2gn" podStartSLOduration=1.244398506 podStartE2EDuration="3.2365945s" totalImagesPullingTime="1.992195994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:00.233898253 +0000 UTC m=+266.463452668" watchObservedRunningTime="2026-10-31 22:54:00.2365945 +0000 UTC m=+266.466148932" Oct 31 22:54:00.298954 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:00.298855 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:00.636022 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:00.635930 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:00.804751 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:00.804715 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:01.218065 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:01.218031 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" event={"ID":"485a8e4b-a74c-4463-affc-44a6374fb81c","Type":"ContainerStarted","Data":"84bd33db9643630363cebf3fecad3b9b291905048da6f3b662d0d8deb5aa49fd"} Oct 31 22:54:01.261868 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:01.261812 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podStartSLOduration=1.499341601 podStartE2EDuration="3.261797995s" totalImagesPullingTime="1.762456394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:01.259028728 +0000 UTC m=+267.488583139" watchObservedRunningTime="2026-10-31 22:54:01.261797995 +0000 UTC m=+267.491352405" Oct 31 22:54:01.928727 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:01.928668 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:02.078042 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:02.078007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:02.222021 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:02.221926 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:54:03.881420 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:03.881378 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:04.133193 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:04.133112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:04.578404 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:04.578350 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:04.725142 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:04.725110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:05.502786 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:05.502743 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:06.757741 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:06.757707 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:09.103623 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.103585 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:09.249286 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.249256 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:09.979599 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.979567 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4"] Oct 31 22:54:09.983245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.983228 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:09.991775 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.991750 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 22:54:09.991895 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.991762 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 22:54:09.991895 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.991852 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ww9gs\"" Oct 31 22:54:09.993235 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:09.993209 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4"] Oct 31 22:54:10.077376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.077342 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdefa176-e223-4de2-857d-14d6ffecf8a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qpdv4\" (UID: \"cdefa176-e223-4de2-857d-14d6ffecf8a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.077575 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.077391 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw988\" (UniqueName: \"kubernetes.io/projected/cdefa176-e223-4de2-857d-14d6ffecf8a7-kube-api-access-cw988\") pod \"cert-manager-webhook-5dbbbf75cc-qpdv4\" (UID: \"cdefa176-e223-4de2-857d-14d6ffecf8a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.178892 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.178860 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdefa176-e223-4de2-857d-14d6ffecf8a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qpdv4\" (UID: \"cdefa176-e223-4de2-857d-14d6ffecf8a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.179275 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.178904 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw988\" (UniqueName: \"kubernetes.io/projected/cdefa176-e223-4de2-857d-14d6ffecf8a7-kube-api-access-cw988\") pod \"cert-manager-webhook-5dbbbf75cc-qpdv4\" (UID: \"cdefa176-e223-4de2-857d-14d6ffecf8a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.191300 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.191270 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdefa176-e223-4de2-857d-14d6ffecf8a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qpdv4\" (UID: \"cdefa176-e223-4de2-857d-14d6ffecf8a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.191412 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.191393 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw988\" (UniqueName: \"kubernetes.io/projected/cdefa176-e223-4de2-857d-14d6ffecf8a7-kube-api-access-cw988\") pod \"cert-manager-webhook-5dbbbf75cc-qpdv4\" (UID: \"cdefa176-e223-4de2-857d-14d6ffecf8a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.214143 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.214117 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 22:54:10.306052 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.306029 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:10.378034 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.377982 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:10.454329 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:54:10.454292 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:54:10.554117 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.554083 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:10.585150 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:10.585065 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4"] Oct 31 22:54:10.588022 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:54:10.587995 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdefa176_e223_4de2_857d_14d6ffecf8a7.slice/crio-d9b4c517810f933be15937e06d1cc0f34597bcd19b970f471735d4d80e150bbf WatchSource:0}: Error finding container d9b4c517810f933be15937e06d1cc0f34597bcd19b970f471735d4d80e150bbf: Status 404 returned error can't find the container with id d9b4c517810f933be15937e06d1cc0f34597bcd19b970f471735d4d80e150bbf Oct 31 22:54:11.128645 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.128615 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt"] Oct 31 22:54:11.133321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.133299 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.144611 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.144589 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tgrvp\"" Oct 31 22:54:11.146007 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.145982 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt"] Oct 31 22:54:11.187463 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.187435 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wjznt\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.187797 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.187471 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7v9\" (UniqueName: \"kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9\") pod \"kuadrant-operator-controller-manager-678d87766f-wjznt\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.262637 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.262605 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" event={"ID":"cdefa176-e223-4de2-857d-14d6ffecf8a7","Type":"ContainerStarted","Data":"d9b4c517810f933be15937e06d1cc0f34597bcd19b970f471735d4d80e150bbf"} Oct 31 22:54:11.288034 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.288011 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wjznt\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.288164 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.288047 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7v9\" (UniqueName: \"kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9\") pod \"kuadrant-operator-controller-manager-678d87766f-wjznt\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.288418 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.288397 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wjznt\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.300589 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.300561 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7v9\" (UniqueName: \"kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9\") pod \"kuadrant-operator-controller-manager-678d87766f-wjznt\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.445685 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.445613 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:11.732052 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:11.732021 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt"] Oct 31 22:54:11.733806 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:54:11.733771 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791e8a63_e940_4430_8bed_f85c1181b68d.slice/crio-7ccac58cc830878043a3c0163ae39fa93d58e52e3790d346bc17c5664b3b17b3 WatchSource:0}: Error finding container 7ccac58cc830878043a3c0163ae39fa93d58e52e3790d346bc17c5664b3b17b3: Status 404 returned error can't find the container with id 7ccac58cc830878043a3c0163ae39fa93d58e52e3790d346bc17c5664b3b17b3 Oct 31 22:54:12.128903 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:12.128869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:12.270861 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:12.270820 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" event={"ID":"791e8a63-e940-4430-8bed-f85c1181b68d","Type":"ContainerStarted","Data":"7ccac58cc830878043a3c0163ae39fa93d58e52e3790d346bc17c5664b3b17b3"} Oct 31 22:54:13.083453 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:13.083415 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:13.230110 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:13.229426 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 22:54:13.238432 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:13.238403 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:14.217352 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:14.217278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:14.287147 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:14.287108 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" event={"ID":"cdefa176-e223-4de2-857d-14d6ffecf8a7","Type":"ContainerStarted","Data":"df921a44d905ad3447310ca266e1269af283784ac8a88b4c5d237bc01349c26c"} Oct 31 22:54:14.287329 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:14.287243 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:14.309815 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:14.309757 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podStartSLOduration=2.040682603 podStartE2EDuration="5.309740231s" totalImagesPullingTime="3.269057628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:14.308609093 +0000 UTC m=+280.538163504" watchObservedRunningTime="2026-10-31 22:54:14.309740231 +0000 UTC m=+280.539294643" Oct 31 22:54:14.389076 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:14.389043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:15.244264 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.243695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:15.419431 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.419400 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:15.592829 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.592749 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7"] Oct 31 22:54:15.596991 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.596965 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.609665 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.609623 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mf9lt\"" Oct 31 22:54:15.611653 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.611630 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7"] Oct 31 22:54:15.622188 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.622161 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:15.627437 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627407 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f223a32-eac0-491e-ae99-0330a8043011-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627582 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627459 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f223a32-eac0-491e-ae99-0330a8043011-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627582 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627505 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627582 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627533 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627813 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627588 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627813 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627613 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p76h\" (UniqueName: \"kubernetes.io/projected/0f223a32-eac0-491e-ae99-0330a8043011-kube-api-access-8p76h\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627813 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627653 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627813 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627706 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.627813 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.627744 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f223a32-eac0-491e-ae99-0330a8043011-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728286 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728262 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728403 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728293 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728403 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728312 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728403 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728330 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p76h\" (UniqueName: \"kubernetes.io/projected/0f223a32-eac0-491e-ae99-0330a8043011-kube-api-access-8p76h\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728403 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728356 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728621 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728416 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728621 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728474 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f223a32-eac0-491e-ae99-0330a8043011-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728621 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728509 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f223a32-eac0-491e-ae99-0330a8043011-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728621 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728581 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f223a32-eac0-491e-ae99-0330a8043011-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728822 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728679 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728822 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728809 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728925 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728884 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.728925 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.728885 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.729354 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.729332 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f223a32-eac0-491e-ae99-0330a8043011-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.731037 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.731014 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f223a32-eac0-491e-ae99-0330a8043011-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.731589 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.731571 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f223a32-eac0-491e-ae99-0330a8043011-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.740798 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.740779 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p76h\" (UniqueName: \"kubernetes.io/projected/0f223a32-eac0-491e-ae99-0330a8043011-kube-api-access-8p76h\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.741135 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.741115 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f223a32-eac0-491e-ae99-0330a8043011-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7\" (UID: \"0f223a32-eac0-491e-ae99-0330a8043011\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.764875 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.764850 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:15.915807 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.915706 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:15.961425 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:15.961378 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:16.216418 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.216388 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7"] Oct 31 22:54:16.219172 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:54:16.219145 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f223a32_eac0_491e_ae99_0330a8043011.slice/crio-10c2df268941612d7024f1795cb4143c294333a5c51084bade01a612844c375a WatchSource:0}: Error finding container 10c2df268941612d7024f1795cb4143c294333a5c51084bade01a612844c375a: Status 404 returned error can't find the container with id 10c2df268941612d7024f1795cb4143c294333a5c51084bade01a612844c375a Oct 31 22:54:16.221282 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.221248 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:54:16.221379 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.221325 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:54:16.221441 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.221395 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:54:16.298983 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.298948 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" event={"ID":"0f223a32-eac0-491e-ae99-0330a8043011","Type":"ContainerStarted","Data":"cc779de7fc8f402601c8bd128bce6cf0d48b369869157fa6075e7b60c3048d16"} Oct 31 22:54:16.299401 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.299006 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" event={"ID":"0f223a32-eac0-491e-ae99-0330a8043011","Type":"ContainerStarted","Data":"10c2df268941612d7024f1795cb4143c294333a5c51084bade01a612844c375a"} Oct 31 22:54:16.300731 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.300708 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" event={"ID":"791e8a63-e940-4430-8bed-f85c1181b68d","Type":"ContainerStarted","Data":"d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810"} Oct 31 22:54:16.300872 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.300783 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:16.351862 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.351816 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" podStartSLOduration=1.351801442 podStartE2EDuration="1.351801442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:16.325624252 +0000 UTC m=+282.555178664" watchObservedRunningTime="2026-10-31 22:54:16.351801442 +0000 UTC m=+282.581355850" Oct 31 22:54:16.352728 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.352696 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" podStartSLOduration=1.367361361 podStartE2EDuration="5.352687615s" totalImagesPullingTime="3.985326254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:16.351122056 +0000 UTC m=+282.580676468" watchObservedRunningTime="2026-10-31 22:54:16.352687615 +0000 UTC m=+282.582242026" Oct 31 22:54:16.916937 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:16.916902 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:17.103925 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.103888 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:17.320814 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.320779 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:17.347245 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.347220 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf"] Oct 31 22:54:17.351001 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.350982 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.364083 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.364061 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-78w28\"" Oct 31 22:54:17.365452 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.365425 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf"] Oct 31 22:54:17.376477 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.376454 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 22:54:17.376601 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.376559 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 22:54:17.382185 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.382165 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 22:54:17.443178 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.443152 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/677d053b-fbf4-4b76-9782-6b3235dc340f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.443271 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.443182 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/677d053b-fbf4-4b76-9782-6b3235dc340f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.443321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.443271 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwcj\" (UniqueName: \"kubernetes.io/projected/677d053b-fbf4-4b76-9782-6b3235dc340f-kube-api-access-ppwcj\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.443321 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.443306 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/677d053b-fbf4-4b76-9782-6b3235dc340f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.465826 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.465802 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:17.543970 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.543938 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/677d053b-fbf4-4b76-9782-6b3235dc340f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.544101 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.543996 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/677d053b-fbf4-4b76-9782-6b3235dc340f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.544101 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.544020 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/677d053b-fbf4-4b76-9782-6b3235dc340f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.544101 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.544087 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwcj\" (UniqueName: \"kubernetes.io/projected/677d053b-fbf4-4b76-9782-6b3235dc340f-kube-api-access-ppwcj\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.544376 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.544358 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/677d053b-fbf4-4b76-9782-6b3235dc340f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.544414 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.544370 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/677d053b-fbf4-4b76-9782-6b3235dc340f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.545038 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.545022 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/677d053b-fbf4-4b76-9782-6b3235dc340f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.556485 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.556466 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwcj\" (UniqueName: \"kubernetes.io/projected/677d053b-fbf4-4b76-9782-6b3235dc340f-kube-api-access-ppwcj\") pod \"kserve-module-controller-manager-7566cf5fc8-4vgvf\" (UID: \"677d053b-fbf4-4b76-9782-6b3235dc340f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.619027 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.618957 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:17.663850 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.663825 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:17.921315 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.921246 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:17.942235 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:17.942212 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf"] Oct 31 22:54:17.944696 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:54:17.944667 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677d053b_fbf4_4b76_9782_6b3235dc340f.slice/crio-1af52a4eec97b94d056c78d3e482533c20f884e24daf5ff704e7ffa2a68b01e8 WatchSource:0}: Error finding container 1af52a4eec97b94d056c78d3e482533c20f884e24daf5ff704e7ffa2a68b01e8: Status 404 returned error can't find the container with id 1af52a4eec97b94d056c78d3e482533c20f884e24daf5ff704e7ffa2a68b01e8 Oct 31 22:54:18.311074 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:18.311036 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" event={"ID":"677d053b-fbf4-4b76-9782-6b3235dc340f","Type":"ContainerStarted","Data":"1af52a4eec97b94d056c78d3e482533c20f884e24daf5ff704e7ffa2a68b01e8"} Oct 31 22:54:18.311342 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:18.311321 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:18.312162 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:18.312140 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" Oct 31 22:54:20.295042 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:20.295019 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 22:54:20.323857 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:20.323829 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" event={"ID":"677d053b-fbf4-4b76-9782-6b3235dc340f","Type":"ContainerStarted","Data":"6b4da7546ae8e32d93087373e955abf2d278c3d6b647180e4b05f83dc7b165ff"} Oct 31 22:54:20.324107 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:20.324090 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:20.353340 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:20.350668 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podStartSLOduration=1.052865592 podStartE2EDuration="3.350650811s" totalImagesPullingTime="2.297785219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:20.350455573 +0000 UTC m=+286.580009986" watchObservedRunningTime="2026-10-31 22:54:20.350650811 +0000 UTC m=+286.580205225" Oct 31 22:54:21.879891 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:21.879855 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:22.080390 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:22.080355 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:22.234944 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:22.234872 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:23.043339 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:23.043305 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:26.089370 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:26.089340 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:26.229429 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:26.229396 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:27.307585 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:27.307538 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 22:54:32.730289 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:32.730254 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:32.876027 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:32.875992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:33.956358 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:33.956329 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:54:33.957409 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:33.957381 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:54:35.923414 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:35.923364 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:36.060918 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:36.060882 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:40.393646 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:40.393595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:40.454160 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:54:40.454126 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:54:40.534267 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:40.534228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:40.734368 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:40.734293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:41.010025 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.009995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:41.145975 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.145917 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:41.323429 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.323357 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p"] Oct 31 22:54:41.332135 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.332112 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 22:54:41.332304 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.332271 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.334958 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.334934 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p"] Oct 31 22:54:41.339496 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.339475 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 22:54:41.339654 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.339634 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-w8h6v\"" Oct 31 22:54:41.339723 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.339523 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 22:54:41.339723 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.339559 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 22:54:41.339723 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.339622 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 22:54:41.339723 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.339518 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 22:54:41.440636 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.440603 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k9j4\" (UniqueName: \"kubernetes.io/projected/863f9cb3-dbae-4b56-b240-eb661de1f2dc-kube-api-access-6k9j4\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.441012 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.440646 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/863f9cb3-dbae-4b56-b240-eb661de1f2dc-cert\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.441012 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.440730 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/863f9cb3-dbae-4b56-b240-eb661de1f2dc-manager-config\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.441012 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.440774 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863f9cb3-dbae-4b56-b240-eb661de1f2dc-metrics-certs\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.541522 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.541491 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/863f9cb3-dbae-4b56-b240-eb661de1f2dc-cert\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.541680 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.541541 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/863f9cb3-dbae-4b56-b240-eb661de1f2dc-manager-config\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.541680 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.541617 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863f9cb3-dbae-4b56-b240-eb661de1f2dc-metrics-certs\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.541793 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.541675 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k9j4\" (UniqueName: \"kubernetes.io/projected/863f9cb3-dbae-4b56-b240-eb661de1f2dc-kube-api-access-6k9j4\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.542226 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.542200 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/863f9cb3-dbae-4b56-b240-eb661de1f2dc-manager-config\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.543940 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.543918 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863f9cb3-dbae-4b56-b240-eb661de1f2dc-metrics-certs\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.544103 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.544079 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/863f9cb3-dbae-4b56-b240-eb661de1f2dc-cert\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.554624 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.554600 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k9j4\" (UniqueName: \"kubernetes.io/projected/863f9cb3-dbae-4b56-b240-eb661de1f2dc-kube-api-access-6k9j4\") pod \"jobset-controller-manager-64f549b4b7-q5w9p\" (UID: \"863f9cb3-dbae-4b56-b240-eb661de1f2dc\") " pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.647340 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.647281 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:41.924309 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.924281 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p"] Oct 31 22:54:41.926307 ip-10-0-133-222 kubenswrapper[5043]: W1031 22:54:41.926275 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863f9cb3_dbae_4b56_b240_eb661de1f2dc.slice/crio-e03edb30ce22700ae9923ce55f4bf4117d390e00790dba66ccfc4888bec34dd6 WatchSource:0}: Error finding container e03edb30ce22700ae9923ce55f4bf4117d390e00790dba66ccfc4888bec34dd6: Status 404 returned error can't find the container with id e03edb30ce22700ae9923ce55f4bf4117d390e00790dba66ccfc4888bec34dd6 Oct 31 22:54:41.928274 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:41.928255 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:54:42.436599 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:42.436561 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" event={"ID":"863f9cb3-dbae-4b56-b240-eb661de1f2dc","Type":"ContainerStarted","Data":"e03edb30ce22700ae9923ce55f4bf4117d390e00790dba66ccfc4888bec34dd6"} Oct 31 22:54:45.528257 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:45.528175 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:45.666340 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:45.666298 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:54:46.459802 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:46.459768 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" event={"ID":"863f9cb3-dbae-4b56-b240-eb661de1f2dc","Type":"ContainerStarted","Data":"8802685cf1bf4a2939a3fd6d39db6ca13a83ed567d0f9ce0cffe32361a521f87"} Oct 31 22:54:46.460020 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:46.459865 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:54:46.481144 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:46.481095 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podStartSLOduration=1.5366669979999998 podStartE2EDuration="5.481080628s" totalImagesPullingTime="3.94441363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:54:46.479406065 +0000 UTC m=+312.708960477" watchObservedRunningTime="2026-10-31 22:54:46.481080628 +0000 UTC m=+312.710635039" Oct 31 22:54:57.469409 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:54:57.469372 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 22:55:07.785873 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:07.785837 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:07.933969 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:07.932522 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:10.252799 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:10.252758 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:10.419065 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:10.419032 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:10.456353 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:55:10.456320 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:55:10.563192 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:10.563105 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:10.715229 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:10.715195 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:11.581217 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:11.581182 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:11.731484 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:11.731452 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:11.941141 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:11.941064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:23.120605 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:23.120572 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:40.454568 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:55:40.454511 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:55:54.739063 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:54.739031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:55:54.887238 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:55:54.887207 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:56:10.455284 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:56:10.455245 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:56:23.194961 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:56:23.194873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:56:40.454786 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:56:40.454744 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:57:10.454009 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:57:10.453962 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:57:23.224314 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:57:23.224275 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:57:40.455210 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:57:40.455168 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:58:10.454293 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:58:10.454210 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:58:13.206151 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:58:13.206123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:58:23.271774 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:58:23.271736 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:58:27.052266 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:58:27.052215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:58:27.210920 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:58:27.210865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:58:38.736625 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:58:38.736579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:58:40.455114 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:58:40.455074 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:59:10.454862 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:59:10.454822 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 22:59:23.418216 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:59:23.418112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:59:33.986757 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:59:33.986722 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:59:33.988634 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:59:33.988614 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 22:59:36.169372 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:59:36.169333 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:59:36.313985 ip-10-0-133-222 kubenswrapper[5043]: I1031 22:59:36.313958 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 22:59:40.454205 ip-10-0-133-222 kubenswrapper[5043]: E1031 22:59:40.454171 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:00:10.454412 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:00:10.454367 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:00:23.370925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:00:23.370887 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:00:40.454359 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:00:40.454313 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:00:46.486078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:00:46.485984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:00:46.651314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:00:46.651273 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:01:10.454272 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:01:10.454235 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:01:23.434127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:01:23.434087 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:01:40.454007 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:01:40.453960 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:02:00.672344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:02:00.672309 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:02:10.454726 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:02:10.454686 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:02:23.487828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:02:23.487751 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:02:40.454219 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:02:40.454184 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:03:10.454241 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:03:10.454207 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:03:23.539110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:03:23.539079 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:03:40.454057 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:03:40.454014 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:04:10.454155 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:04:10.454117 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:04:23.678928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:23.678891 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:04:34.016166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:34.016133 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:04:34.021405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:34.021383 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:04:36.336976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.336945 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wjznt_791e8a63-e940-4430-8bed-f85c1181b68d/manager/0.log" Oct 31 23:04:36.535905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.535872 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt"] Oct 31 23:04:36.536126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.536102 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" podUID="791e8a63-e940-4430-8bed-f85c1181b68d" containerName="manager" containerID="cri-o://d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810" gracePeriod=10 Oct 31 23:04:36.570112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.570089 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8"] Oct 31 23:04:36.573526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.573510 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.587219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.587159 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8"] Oct 31 23:04:36.645079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.645051 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xhfq8\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.645198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.645112 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9967d\" (UniqueName: \"kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d\") pod \"kuadrant-operator-controller-manager-678d87766f-xhfq8\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.746123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.746089 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xhfq8\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.746255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.746137 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9967d\" (UniqueName: \"kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d\") pod \"kuadrant-operator-controller-manager-678d87766f-xhfq8\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.746450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.746429 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xhfq8\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.760870 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.760850 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9967d\" (UniqueName: \"kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d\") pod \"kuadrant-operator-controller-manager-678d87766f-xhfq8\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.778895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.778878 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 23:04:36.847366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.847299 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9\" (UniqueName: \"kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9\") pod \"791e8a63-e940-4430-8bed-f85c1181b68d\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " Oct 31 23:04:36.847480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.847357 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume\") pod \"791e8a63-e940-4430-8bed-f85c1181b68d\" (UID: \"791e8a63-e940-4430-8bed-f85c1181b68d\") " Oct 31 23:04:36.847775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.847749 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume" pod "791e8a63-e940-4430-8bed-f85c1181b68d" (UID: "791e8a63-e940-4430-8bed-f85c1181b68d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:04:36.849380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.849354 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9" pod "791e8a63-e940-4430-8bed-f85c1181b68d" (UID: "791e8a63-e940-4430-8bed-f85c1181b68d"). InnerVolumeSpecName "kube-api-access-kx7v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:04:36.924291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.924254 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:36.948883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.948838 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7v9\" (UniqueName: \"kubernetes.io/projected/791e8a63-e940-4430-8bed-f85c1181b68d-kube-api-access-kx7v9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:04:36.948998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:36.948954 5043 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/791e8a63-e940-4430-8bed-f85c1181b68d-extensions-socket-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:04:37.212438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.212350 5043 generic.go:356] "Generic (PLEG): container finished" podID="791e8a63-e940-4430-8bed-f85c1181b68d" containerID="d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810" exitCode=0 Oct 31 23:04:37.212605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.212440 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" Oct 31 23:04:37.212605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.212433 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" event={"ID":"791e8a63-e940-4430-8bed-f85c1181b68d","Type":"ContainerDied","Data":"d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810"} Oct 31 23:04:37.212605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.212596 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt" event={"ID":"791e8a63-e940-4430-8bed-f85c1181b68d","Type":"ContainerDied","Data":"7ccac58cc830878043a3c0163ae39fa93d58e52e3790d346bc17c5664b3b17b3"} Oct 31 23:04:37.212729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.212621 5043 scope.go:122] "RemoveContainer" containerID="d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810" Oct 31 23:04:37.218847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.218803 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8"] Oct 31 23:04:37.223491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.223468 5043 scope.go:122] "RemoveContainer" containerID="d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810" Oct 31 23:04:37.223822 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:04:37.223801 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810\": container with ID starting with d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810 not found: ID does not exist" containerID="d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810" Oct 31 23:04:37.223899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.223831 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810"} err="failed to get container status \"d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810\": rpc error: code = NotFound desc = could not find container \"d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810\": container with ID starting with d5fa242e4dc089b29aadcdcf312addf5f55272f297e1df8ce860a0228fe60810 not found: ID does not exist" Oct 31 23:04:37.224767 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:04:37.224745 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a050c1_60b2_4ee9_a860_8f646427645e.slice/crio-fefc23149435e4d991a70cb2f65578924e9d259c4c31496b4ca039a70430355c WatchSource:0}: Error finding container fefc23149435e4d991a70cb2f65578924e9d259c4c31496b4ca039a70430355c: Status 404 returned error can't find the container with id fefc23149435e4d991a70cb2f65578924e9d259c4c31496b4ca039a70430355c Oct 31 23:04:37.247510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.247487 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt"] Oct 31 23:04:37.253284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.253263 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wjznt"] Oct 31 23:04:37.934275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:37.934243 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791e8a63-e940-4430-8bed-f85c1181b68d" path="/var/lib/kubelet/pods/791e8a63-e940-4430-8bed-f85c1181b68d/volumes" Oct 31 23:04:38.219665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:38.219584 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" event={"ID":"e0a050c1-60b2-4ee9-a860-8f646427645e","Type":"ContainerStarted","Data":"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916"} Oct 31 23:04:38.219665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:38.219640 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" event={"ID":"e0a050c1-60b2-4ee9-a860-8f646427645e","Type":"ContainerStarted","Data":"fefc23149435e4d991a70cb2f65578924e9d259c4c31496b4ca039a70430355c"} Oct 31 23:04:38.219847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:38.219723 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:38.245502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:38.245453 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podStartSLOduration=2.245441825 podStartE2EDuration="2.245441825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:04:38.245039916 +0000 UTC m=+904.474594327" watchObservedRunningTime="2026-10-31 23:04:38.245441825 +0000 UTC m=+904.474996235" Oct 31 23:04:40.453982 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:04:40.453946 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:04:49.226677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:49.226649 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:04:53.369088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:53.369043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:04:54.220493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.220460 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jdtjv"] Oct 31 23:04:54.220828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.220816 5043 cpu_manager.go:414] "containerMap: removing container" podUID="791e8a63-e940-4430-8bed-f85c1181b68d" containerName="manager" Oct 31 23:04:54.220865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.220830 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791e8a63-e940-4430-8bed-f85c1181b68d" containerName="manager" Oct 31 23:04:54.220916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.220891 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="791e8a63-e940-4430-8bed-f85c1181b68d" containerName="manager" Oct 31 23:04:54.224036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.224016 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:04:54.229831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.229810 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p44kh\"" Oct 31 23:04:54.230826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.230804 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jdtjv"] Oct 31 23:04:54.290278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.290249 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dns\" (UniqueName: \"kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns\") pod \"authorino-56c6c9ddd5-jdtjv\" (UID: \"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c\") " pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:04:54.391068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.391029 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25dns\" (UniqueName: \"kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns\") pod \"authorino-56c6c9ddd5-jdtjv\" (UID: \"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c\") " pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:04:54.403932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.403910 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dns\" (UniqueName: \"kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns\") pod \"authorino-56c6c9ddd5-jdtjv\" (UID: \"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c\") " pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:04:54.536783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.536759 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:04:54.631199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.631171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:04:54.810741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.810711 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jdtjv"] Oct 31 23:04:54.813387 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:04:54.813359 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ad2fd6d_4faa_4ac1_a908_808f180c9e8c.slice/crio-97dbbdb02b2cc4d436757931be5546a5c34f61b6eb2f112e6af6d4888372b9c3 WatchSource:0}: Error finding container 97dbbdb02b2cc4d436757931be5546a5c34f61b6eb2f112e6af6d4888372b9c3: Status 404 returned error can't find the container with id 97dbbdb02b2cc4d436757931be5546a5c34f61b6eb2f112e6af6d4888372b9c3 Oct 31 23:04:54.814631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:54.814614 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:04:55.304271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:55.304238 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" event={"ID":"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c","Type":"ContainerStarted","Data":"97dbbdb02b2cc4d436757931be5546a5c34f61b6eb2f112e6af6d4888372b9c3"} Oct 31 23:04:57.316056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:57.316018 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" event={"ID":"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c","Type":"ContainerStarted","Data":"17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3"} Oct 31 23:04:57.335111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:04:57.335053 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" podStartSLOduration=1.692092095 podStartE2EDuration="3.335036107s" totalImagesPullingTime="1.642944012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:04:57.333775479 +0000 UTC m=+923.563329911" watchObservedRunningTime="2026-10-31 23:04:57.335036107 +0000 UTC m=+923.564590520" Oct 31 23:05:10.453923 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:05:10.453885 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:05:11.249843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:11.249809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:11.394296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:11.394266 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:12.087246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:12.087213 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:23.661870 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:23.661795 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:31.694287 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:31.694243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:31.856162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:31.856130 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:31.994310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:31.994277 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:35.395949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:35.395914 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:35.586420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:35.586388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:35.935413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:35.935374 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:36.100805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:36.100771 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:36.970442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:36.970410 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:37.106081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:37.106048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:05:40.454862 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:05:40.454828 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:05:52.731158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:05:52.731098 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:07.414982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:07.414917 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:07.579398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:07.579365 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:08.713448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:08.713416 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:08.856276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:08.856243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:09.000933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:09.000905 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:10.456314 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:06:10.456263 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:06:19.110426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.110370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:19.255074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.255046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:19.429739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.429707 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ksq72"] Oct 31 23:06:19.433106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.433085 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.440809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.440788 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ksq72"] Oct 31 23:06:19.441849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.441835 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 23:06:19.610377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.610299 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert\") pod \"authorino-6889cbd8c4-ksq72\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.610377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.610350 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drnd\" (UniqueName: \"kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd\") pod \"authorino-6889cbd8c4-ksq72\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.693947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.693919 5043 generic.go:356] "Generic (PLEG): container finished" podID="11dc44da-2acd-4370-a0c7-37b534779995" containerID="c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3" exitCode=0 Oct 31 23:06:19.694103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.693998 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" event={"ID":"11dc44da-2acd-4370-a0c7-37b534779995","Type":"ContainerDied","Data":"c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3"} Oct 31 23:06:19.694365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.694348 5043 scope.go:122] "RemoveContainer" containerID="c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3" Oct 31 23:06:19.695405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.695378 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:06:19.695481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.695443 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:06:19.695481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.695474 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:06:19.711369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.711346 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert\") pod \"authorino-6889cbd8c4-ksq72\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.711481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.711407 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drnd\" (UniqueName: \"kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd\") pod \"authorino-6889cbd8c4-ksq72\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.713735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.713715 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert\") pod \"authorino-6889cbd8c4-ksq72\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.728302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.728280 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drnd\" (UniqueName: \"kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd\") pod \"authorino-6889cbd8c4-ksq72\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.746209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.746184 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:19.756666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:19.756523 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ksq72"] Oct 31 23:06:20.064510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:20.063010 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ksq72"] Oct 31 23:06:20.076315 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:06:20.076122 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63ce4ef7_b88d_48ac_8dc3_b52e0ef565fa.slice/crio-139ede63bb09f3838876bd64d05ca4a85648208b11b79223b3d496560a586466 WatchSource:0}: Error finding container 139ede63bb09f3838876bd64d05ca4a85648208b11b79223b3d496560a586466: Status 404 returned error can't find the container with id 139ede63bb09f3838876bd64d05ca4a85648208b11b79223b3d496560a586466 Oct 31 23:06:20.700650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:20.700616 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" event={"ID":"11dc44da-2acd-4370-a0c7-37b534779995","Type":"ContainerStarted","Data":"240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436"} Oct 31 23:06:20.701874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:20.701848 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" event={"ID":"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa","Type":"ContainerStarted","Data":"139ede63bb09f3838876bd64d05ca4a85648208b11b79223b3d496560a586466"} Oct 31 23:06:21.668578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.666771 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 23:06:21.668578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.666893 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 31 23:06:21.668578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.666933 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 31 23:06:21.710750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.710697 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" event={"ID":"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa","Type":"ContainerStarted","Data":"803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757"} Oct 31 23:06:21.711217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.710744 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" podUID="63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" containerName="authorino" containerID="cri-o://803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757" gracePeriod=30 Oct 31 23:06:21.737405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.736364 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jdtjv"] Oct 31 23:06:21.737405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.736577 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" podUID="8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" containerName="authorino" containerID="cri-o://17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3" gracePeriod=30 Oct 31 23:06:21.772087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:21.772046 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" podStartSLOduration=1.962861666 podStartE2EDuration="2.772030287s" totalImagesPullingTime="809.168621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:06:21.752454708 +0000 UTC m=+1007.982009117" watchObservedRunningTime="2026-10-31 23:06:21.772030287 +0000 UTC m=+1008.001584967" Oct 31 23:06:22.052650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.052622 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:22.063688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.063662 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd\" (UniqueName: \"kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd\") pod \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " Oct 31 23:06:22.063816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.063780 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert\" (UniqueName: \"kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert\") pod \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\" (UID: \"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa\") " Oct 31 23:06:22.065691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.065662 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd" pod "63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" (UID: "63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa"). InnerVolumeSpecName "kube-api-access-4drnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:06:22.069164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.069143 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:06:22.076280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.076257 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert" pod "63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" (UID: "63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:06:22.164953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.164924 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns\" (UniqueName: \"kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns\") pod \"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c\" (UID: \"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c\") " Oct 31 23:06:22.165121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.165081 5043 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-tls-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:22.165121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.165095 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drnd\" (UniqueName: \"kubernetes.io/projected/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa-kube-api-access-4drnd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:22.166932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.166906 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns" pod "8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" (UID: "8ad2fd6d-4faa-4ac1-a908-808f180c9e8c"). InnerVolumeSpecName "kube-api-access-25dns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:06:22.266489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.266457 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25dns\" (UniqueName: \"kubernetes.io/projected/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c-kube-api-access-25dns\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:06:22.665916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.665839 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 31 23:06:22.665916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.665889 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 31 23:06:22.732579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.728037 5043 generic.go:356] "Generic (PLEG): container finished" podID="63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" containerID="803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757" exitCode=0 Oct 31 23:06:22.732579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.728195 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" event={"ID":"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa","Type":"ContainerDied","Data":"803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757"} Oct 31 23:06:22.732579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.728234 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" event={"ID":"63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa","Type":"ContainerDied","Data":"139ede63bb09f3838876bd64d05ca4a85648208b11b79223b3d496560a586466"} Oct 31 23:06:22.732579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.728258 5043 scope.go:122] "RemoveContainer" containerID="803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757" Oct 31 23:06:22.732579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.728401 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-ksq72" Oct 31 23:06:22.740394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.740368 5043 generic.go:356] "Generic (PLEG): container finished" podID="8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" containerID="17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3" exitCode=0 Oct 31 23:06:22.740495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.740426 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" Oct 31 23:06:22.740495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.740459 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" event={"ID":"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c","Type":"ContainerDied","Data":"17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3"} Oct 31 23:06:22.740653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.740520 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jdtjv" event={"ID":"8ad2fd6d-4faa-4ac1-a908-808f180c9e8c","Type":"ContainerDied","Data":"97dbbdb02b2cc4d436757931be5546a5c34f61b6eb2f112e6af6d4888372b9c3"} Oct 31 23:06:22.748055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.748039 5043 scope.go:122] "RemoveContainer" containerID="803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757" Oct 31 23:06:22.748331 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:06:22.748310 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757\": container with ID starting with 803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757 not found: ID does not exist" containerID="803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757" Oct 31 23:06:22.748401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.748337 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757"} err="failed to get container status \"803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757\": rpc error: code = NotFound desc = could not find container \"803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757\": container with ID starting with 803d8298670ca63498f5ac05742a24d88807ba8a971b9adba52a12a4f91ef757 not found: ID does not exist" Oct 31 23:06:22.748401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.748353 5043 scope.go:122] "RemoveContainer" containerID="17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3" Oct 31 23:06:22.756338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.756325 5043 scope.go:122] "RemoveContainer" containerID="17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3" Oct 31 23:06:22.756588 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:06:22.756568 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3\": container with ID starting with 17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3 not found: ID does not exist" containerID="17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3" Oct 31 23:06:22.756676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.756593 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3"} err="failed to get container status \"17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3\": rpc error: code = NotFound desc = could not find container \"17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3\": container with ID starting with 17d04b2a62b6c617abbd76ecca5df1b8604b83380e6932c80924d03cc84e2bd3 not found: ID does not exist" Oct 31 23:06:22.761334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.761314 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ksq72"] Oct 31 23:06:22.765274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.765255 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-ksq72"] Oct 31 23:06:22.776783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.776766 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jdtjv"] Oct 31 23:06:22.781080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.781062 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jdtjv"] Oct 31 23:06:22.990541 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:22.990461 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 23:06:23.664946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:23.664914 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 31 23:06:23.665162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:23.664971 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 31 23:06:23.711524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:23.711497 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:23.945677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:23.945605 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" path="/var/lib/kubelet/pods/63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa/volumes" Oct 31 23:06:23.946024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:23.945930 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" path="/var/lib/kubelet/pods/8ad2fd6d-4faa-4ac1-a908-808f180c9e8c/volumes" Oct 31 23:06:24.665559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:24.665523 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 31 23:06:24.665704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:24.665587 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 31 23:06:25.669237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:25.669204 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 23:06:25.670177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:25.670160 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 23:06:40.454386 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:06:40.454351 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:06:56.256741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:56.256703 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:06:56.553595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:06:56.553491 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:10.455654 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:07:10.455611 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:07:12.945064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:12.945030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:15.388146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388117 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g"] Oct 31 23:07:15.388603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388484 5043 cpu_manager.go:414] "containerMap: removing container" podUID="63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" containerName="authorino" Oct 31 23:07:15.388603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388495 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" containerName="authorino" Oct 31 23:07:15.388603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388506 5043 cpu_manager.go:414] "containerMap: removing container" podUID="8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" containerName="authorino" Oct 31 23:07:15.388603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388513 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" containerName="authorino" Oct 31 23:07:15.388897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388656 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ce4ef7-b88d-48ac-8dc3-b52e0ef565fa" containerName="authorino" Oct 31 23:07:15.388897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.388671 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad2fd6d-4faa-4ac1-a908-808f180c9e8c" containerName="authorino" Oct 31 23:07:15.391917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.391898 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.396380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.396357 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 23:07:15.397661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.397629 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ltzbh\"" Oct 31 23:07:15.397746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.397673 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 23:07:15.397746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.397734 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 23:07:15.401649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.401627 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g"] Oct 31 23:07:15.475460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.475437 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:15.498757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.498735 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzqs\" (UniqueName: \"kubernetes.io/projected/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-kube-api-access-clzqs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.498862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.498776 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.498862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.498850 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.498939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.498873 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.498939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.498895 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.498939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.498915 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600095 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600135 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600161 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600185 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600212 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzqs\" (UniqueName: \"kubernetes.io/projected/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-kube-api-access-clzqs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600582 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600624 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.600741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.600722 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.602474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.602451 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.602799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.602780 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.613013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.612984 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzqs\" (UniqueName: \"kubernetes.io/projected/f091b8de-14bb-4d9d-ac75-3edd31cb3e68-kube-api-access-clzqs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-2859g\" (UID: \"f091b8de-14bb-4d9d-ac75-3edd31cb3e68\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:15.706501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:15.706414 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:16.003092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:16.003064 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g"] Oct 31 23:07:16.005264 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:07:16.005239 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf091b8de_14bb_4d9d_ac75_3edd31cb3e68.slice/crio-a8c2dc469ffaa988e3c5e3b3727f5e09ec9ddb412c831d8b8115d76680788f9a WatchSource:0}: Error finding container a8c2dc469ffaa988e3c5e3b3727f5e09ec9ddb412c831d8b8115d76680788f9a: Status 404 returned error can't find the container with id a8c2dc469ffaa988e3c5e3b3727f5e09ec9ddb412c831d8b8115d76680788f9a Oct 31 23:07:17.000936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:17.000898 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" event={"ID":"f091b8de-14bb-4d9d-ac75-3edd31cb3e68","Type":"ContainerStarted","Data":"a8c2dc469ffaa988e3c5e3b3727f5e09ec9ddb412c831d8b8115d76680788f9a"} Oct 31 23:07:18.006606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:18.006571 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" event={"ID":"f091b8de-14bb-4d9d-ac75-3edd31cb3e68","Type":"ContainerStarted","Data":"e45d17cce29cc2391df8e2f21721f22b5294d65a86337b6cb6faa1ea4db9773d"} Oct 31 23:07:18.030787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:18.030743 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podStartSLOduration=1.728411948 podStartE2EDuration="3.030729946s" totalImagesPullingTime="1.302317998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:18.029175582 +0000 UTC m=+1064.258729993" watchObservedRunningTime="2026-10-31 23:07:18.030729946 +0000 UTC m=+1064.260284355" Oct 31 23:07:18.309144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:18.309030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:18.706883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:18.706804 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:20.008401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.008371 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-55lxc"] Oct 31 23:07:20.012165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.012139 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.019880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.019528 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 23:07:20.021461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.021436 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-55lxc"] Oct 31 23:07:20.095636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.095599 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:20.148860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.148823 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rh6\" (UniqueName: \"kubernetes.io/projected/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-kube-api-access-b9rh6\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.149024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.148887 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.149024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.148971 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-home\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.149024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.148991 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.149024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.149008 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.149283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.149264 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250350 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250395 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rh6\" (UniqueName: \"kubernetes.io/projected/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-kube-api-access-b9rh6\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250430 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250474 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-home\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250490 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250504 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250882 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.250975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250943 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-home\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.251026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.250982 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.252798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.252779 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.252910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.252894 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.278872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.278815 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rh6\" (UniqueName: \"kubernetes.io/projected/9cf1ba00-1cc7-4af1-a5df-efeafadaac9b-kube-api-access-b9rh6\") pod \"echo-simulator-kserve-64b6f8f96c-55lxc\" (UID: \"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.327536 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.327511 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:20.607251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:20.607164 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-55lxc"] Oct 31 23:07:20.610002 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:07:20.609975 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf1ba00_1cc7_4af1_a5df_efeafadaac9b.slice/crio-c04215a28c4ffe06e07794c6f5af7d91864dd8ea6e2ec6cf2af0de1634c44486 WatchSource:0}: Error finding container c04215a28c4ffe06e07794c6f5af7d91864dd8ea6e2ec6cf2af0de1634c44486: Status 404 returned error can't find the container with id c04215a28c4ffe06e07794c6f5af7d91864dd8ea6e2ec6cf2af0de1634c44486 Oct 31 23:07:21.024279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.024237 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" event={"ID":"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b","Type":"ContainerStarted","Data":"c04215a28c4ffe06e07794c6f5af7d91864dd8ea6e2ec6cf2af0de1634c44486"} Oct 31 23:07:21.084165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.084135 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz"] Oct 31 23:07:21.087562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.087531 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.098920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.098886 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 23:07:21.101165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.101141 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz"] Oct 31 23:07:21.173262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.173209 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:21.258582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.258532 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.258761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.258649 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.258761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.258699 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d63e791b-7230-4884-bdba-3996f4205f60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.258761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.258720 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.258951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.258796 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rrc\" (UniqueName: \"kubernetes.io/projected/d63e791b-7230-4884-bdba-3996f4205f60-kube-api-access-g6rrc\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.258951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.258887 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.360324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.360229 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d63e791b-7230-4884-bdba-3996f4205f60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.360324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.360281 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.360324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.360303 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rrc\" (UniqueName: \"kubernetes.io/projected/d63e791b-7230-4884-bdba-3996f4205f60-kube-api-access-g6rrc\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.360666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.360343 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.360666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.360380 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.360666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.360415 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.361603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.361182 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.361603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.361478 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.362007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.361983 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.363912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.363889 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d63e791b-7230-4884-bdba-3996f4205f60-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.365340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.365297 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d63e791b-7230-4884-bdba-3996f4205f60-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.374050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.374030 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rrc\" (UniqueName: \"kubernetes.io/projected/d63e791b-7230-4884-bdba-3996f4205f60-kube-api-access-g6rrc\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-spffz\" (UID: \"d63e791b-7230-4884-bdba-3996f4205f60\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.406808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.406769 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:21.704264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:21.704236 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz"] Oct 31 23:07:21.705983 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:07:21.705959 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63e791b_7230_4884_bdba_3996f4205f60.slice/crio-516b826db72a748d99b35846f68a3368347318922ef5c9b9fdca3f87cf6e620a WatchSource:0}: Error finding container 516b826db72a748d99b35846f68a3368347318922ef5c9b9fdca3f87cf6e620a: Status 404 returned error can't find the container with id 516b826db72a748d99b35846f68a3368347318922ef5c9b9fdca3f87cf6e620a Oct 31 23:07:22.032404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:22.032352 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" event={"ID":"d63e791b-7230-4884-bdba-3996f4205f60","Type":"ContainerStarted","Data":"516b826db72a748d99b35846f68a3368347318922ef5c9b9fdca3f87cf6e620a"} Oct 31 23:07:22.412071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:22.411990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:23.039586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.039181 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" event={"ID":"9cf1ba00-1cc7-4af1-a5df-efeafadaac9b","Type":"ContainerStarted","Data":"2846e881fb615ae59c2fc94cc393a1abd149f03eb69d33099a0aa6a4e26e73d1"} Oct 31 23:07:23.041152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.041116 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" event={"ID":"d63e791b-7230-4884-bdba-3996f4205f60","Type":"ContainerStarted","Data":"021fd125df52c41c8b506e814283b4d82dc2ecc1869a7c8f5b65e0d89890a7a9"} Oct 31 23:07:23.062187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.062135 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podStartSLOduration=2.070660618 podStartE2EDuration="4.062118062s" totalImagesPullingTime="1.991457444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:23.058831138 +0000 UTC m=+1069.288385549" watchObservedRunningTime="2026-10-31 23:07:23.062118062 +0000 UTC m=+1069.291672473" Oct 31 23:07:23.082857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.082812 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podStartSLOduration=1.7403396789999999 podStartE2EDuration="2.082800195s" totalImagesPullingTime="342.460516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:23.080880888 +0000 UTC m=+1069.310435303" watchObservedRunningTime="2026-10-31 23:07:23.082800195 +0000 UTC m=+1069.312354605" Oct 31 23:07:23.328645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.327932 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:23.391706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.391658 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:23.407499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.407465 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:23.790298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:23.790267 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:24.272172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.272137 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm"] Oct 31 23:07:24.276036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.276010 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.285455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.285432 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 23:07:24.289747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.289721 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm"] Oct 31 23:07:24.291160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.291136 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.291336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.291297 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98438769-91a9-4aec-b9e9-d0c4044cae37-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.291444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.291343 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.291503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.291457 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.291598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.291575 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.291669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.291616 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bjb\" (UniqueName: \"kubernetes.io/projected/98438769-91a9-4aec-b9e9-d0c4044cae37-kube-api-access-k7bjb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.392452 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.392402 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.392655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.392465 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.392655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.392493 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bjb\" (UniqueName: \"kubernetes.io/projected/98438769-91a9-4aec-b9e9-d0c4044cae37-kube-api-access-k7bjb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.392655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.392617 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.393626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.392972 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98438769-91a9-4aec-b9e9-d0c4044cae37-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.393626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.393060 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.394135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.394105 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.394315 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.394258 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.394470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.394442 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.396393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.396369 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98438769-91a9-4aec-b9e9-d0c4044cae37-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.396522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.396393 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98438769-91a9-4aec-b9e9-d0c4044cae37-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.414400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.414345 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bjb\" (UniqueName: \"kubernetes.io/projected/98438769-91a9-4aec-b9e9-d0c4044cae37-kube-api-access-k7bjb\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-ld9pm\" (UID: \"98438769-91a9-4aec-b9e9-d0c4044cae37\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.478022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.477982 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:24.592640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.592568 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:24.887612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:24.887588 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm"] Oct 31 23:07:24.889266 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:07:24.889240 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98438769_91a9_4aec_b9e9_d0c4044cae37.slice/crio-01794460ea0eb73590c3f0e113a098b685af6b0dfd5cc35166e748538cd62980 WatchSource:0}: Error finding container 01794460ea0eb73590c3f0e113a098b685af6b0dfd5cc35166e748538cd62980: Status 404 returned error can't find the container with id 01794460ea0eb73590c3f0e113a098b685af6b0dfd5cc35166e748538cd62980 Oct 31 23:07:25.053608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:25.053077 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" event={"ID":"98438769-91a9-4aec-b9e9-d0c4044cae37","Type":"ContainerStarted","Data":"01794460ea0eb73590c3f0e113a098b685af6b0dfd5cc35166e748538cd62980"} Oct 31 23:07:25.706865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:25.706828 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:25.725130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:25.725101 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:26.060639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:26.060606 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" event={"ID":"98438769-91a9-4aec-b9e9-d0c4044cae37","Type":"ContainerStarted","Data":"98a282324cd585948a995b10a6c6567f1ff23f524def98bed9b51278fff26087"} Oct 31 23:07:26.071895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:26.071870 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:07:26.082998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:26.082954 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podStartSLOduration=1.7680502630000001 podStartE2EDuration="2.082942061s" totalImagesPullingTime="314.891798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:26.082534318 +0000 UTC m=+1072.312088731" watchObservedRunningTime="2026-10-31 23:07:26.082942061 +0000 UTC m=+1072.312496470" Oct 31 23:07:26.592721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:26.592685 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:30.327608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:30.327579 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:30.339939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:30.339910 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:31.097696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:31.097669 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:07:31.407210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:31.407134 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:31.419494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:31.419467 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:32.054156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:32.054127 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:32.101609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:32.101584 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:07:34.593196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:34.593159 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:34.605522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:34.605498 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:35.117058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:35.117028 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:07:39.468010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.467979 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bfb984676-kv9dg"] Oct 31 23:07:39.472830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.472811 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.480887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.480867 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 23:07:39.481031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.480909 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p44kh\"" Oct 31 23:07:39.481694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.481674 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bfb984676-kv9dg"] Oct 31 23:07:39.532162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.532135 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hs2f\" (UniqueName: \"kubernetes.io/projected/d55a9c0b-0e57-4c80-b2e2-929b74863635-kube-api-access-6hs2f\") pod \"authorino-6bfb984676-kv9dg\" (UID: \"d55a9c0b-0e57-4c80-b2e2-929b74863635\") " pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.532281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.532207 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d55a9c0b-0e57-4c80-b2e2-929b74863635-tls-cert\") pod \"authorino-6bfb984676-kv9dg\" (UID: \"d55a9c0b-0e57-4c80-b2e2-929b74863635\") " pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.633481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.633446 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hs2f\" (UniqueName: \"kubernetes.io/projected/d55a9c0b-0e57-4c80-b2e2-929b74863635-kube-api-access-6hs2f\") pod \"authorino-6bfb984676-kv9dg\" (UID: \"d55a9c0b-0e57-4c80-b2e2-929b74863635\") " pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.633661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.633505 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d55a9c0b-0e57-4c80-b2e2-929b74863635-tls-cert\") pod \"authorino-6bfb984676-kv9dg\" (UID: \"d55a9c0b-0e57-4c80-b2e2-929b74863635\") " pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.635939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.635906 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d55a9c0b-0e57-4c80-b2e2-929b74863635-tls-cert\") pod \"authorino-6bfb984676-kv9dg\" (UID: \"d55a9c0b-0e57-4c80-b2e2-929b74863635\") " pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.646182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.646156 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hs2f\" (UniqueName: \"kubernetes.io/projected/d55a9c0b-0e57-4c80-b2e2-929b74863635-kube-api-access-6hs2f\") pod \"authorino-6bfb984676-kv9dg\" (UID: \"d55a9c0b-0e57-4c80-b2e2-929b74863635\") " pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:39.786228 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:39.786198 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bfb984676-kv9dg" Oct 31 23:07:40.063626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:40.063602 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bfb984676-kv9dg"] Oct 31 23:07:40.065661 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:07:40.065635 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd55a9c0b_0e57_4c80_b2e2_929b74863635.slice/crio-23032201471d63cbdd0dacbf6f884b7ebc82b7c8520097613476c20da59dda56 WatchSource:0}: Error finding container 23032201471d63cbdd0dacbf6f884b7ebc82b7c8520097613476c20da59dda56: Status 404 returned error can't find the container with id 23032201471d63cbdd0dacbf6f884b7ebc82b7c8520097613476c20da59dda56 Oct 31 23:07:40.135862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:40.135831 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bfb984676-kv9dg" event={"ID":"d55a9c0b-0e57-4c80-b2e2-929b74863635","Type":"ContainerStarted","Data":"23032201471d63cbdd0dacbf6f884b7ebc82b7c8520097613476c20da59dda56"} Oct 31 23:07:40.454883 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:07:40.454799 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:07:42.148481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:42.148440 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bfb984676-kv9dg" event={"ID":"d55a9c0b-0e57-4c80-b2e2-929b74863635","Type":"ContainerStarted","Data":"bb8f3f12fd7b906674f1df335842837cbc1c0ee446e218d1219413c113dd69d5"} Oct 31 23:07:42.169461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:42.169411 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6bfb984676-kv9dg" podStartSLOduration=2.149786185 podStartE2EDuration="3.169397221s" totalImagesPullingTime="1.019611036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:42.168177162 +0000 UTC m=+1088.397731571" watchObservedRunningTime="2026-10-31 23:07:42.169397221 +0000 UTC m=+1088.398951631" Oct 31 23:07:42.814144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:42.814108 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:42.957230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:42.957192 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:46.527437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:46.527402 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:46.666930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:46.666884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:07:47.314532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.314499 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl"] Oct 31 23:07:47.321039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.321019 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.344938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.344908 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl"] Oct 31 23:07:47.411309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411274 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411309 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411328 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrkj\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411430 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411448 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411466 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411482 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411562 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.411688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.411617 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512359 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512399 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512430 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512454 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512482 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512535 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512631 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512656 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512679 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrkj\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.512961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.512878 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.513068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.513017 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.513224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.513195 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.513326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.513201 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.513326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.513307 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.514883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.514852 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.515943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.515907 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.525447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.525422 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrkj\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.525574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.525536 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-p9wjl\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.635751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.635668 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:47.932725 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:07:47.932685 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796f8080_e834_4e54_aaca_808246fe73e5.slice/crio-abfa7fb36ee336ec31fc3288dd93e6e980c60103dc3530aeffeef432a202608a WatchSource:0}: Error finding container abfa7fb36ee336ec31fc3288dd93e6e980c60103dc3530aeffeef432a202608a: Status 404 returned error can't find the container with id abfa7fb36ee336ec31fc3288dd93e6e980c60103dc3530aeffeef432a202608a Oct 31 23:07:47.934824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.934793 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:07:47.934922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.934854 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:07:47.934922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.934892 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:07:47.935073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:47.935056 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl"] Oct 31 23:07:48.185876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:48.185781 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" event={"ID":"796f8080-e834-4e54-aaca-808246fe73e5","Type":"ContainerStarted","Data":"15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56"} Oct 31 23:07:48.185876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:48.185863 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" event={"ID":"796f8080-e834-4e54-aaca-808246fe73e5","Type":"ContainerStarted","Data":"abfa7fb36ee336ec31fc3288dd93e6e980c60103dc3530aeffeef432a202608a"} Oct 31 23:07:48.211664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:48.211610 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podStartSLOduration=1.211593854 podStartE2EDuration="1.211593854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:07:48.209533489 +0000 UTC m=+1094.439087913" watchObservedRunningTime="2026-10-31 23:07:48.211593854 +0000 UTC m=+1094.441148266" Oct 31 23:07:48.638739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:48.638698 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:48.640458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:48.640432 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:48.640541 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:48.640499 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:49.638980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:49.638938 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:49.639348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:49.639005 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:50.637569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:50.636941 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:50.637569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:50.637018 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:51.638956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:51.638919 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:51.639432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:51.638977 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:52.638974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:52.638937 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:52.639371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:52.639001 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:53.636900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:53.636866 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:53.637072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:53.636937 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:54.636684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:54.636650 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 23:07:54.637045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:54.636711 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 23:07:55.854230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:55.854198 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:55.854702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:55.854595 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:07:55.855515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:07:55.855495 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:08:02.291013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.290978 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb"] Oct 31 23:08:02.298176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.298153 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.307404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.307372 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb"] Oct 31 23:08:02.464730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464692 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.464730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464731 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.464946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464756 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.464946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464815 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvr8\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.464946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464891 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.465040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464962 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.465040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.464998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.465040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.465013 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.465132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.465108 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566054 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566090 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566115 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566139 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvr8\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566162 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566187 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566233 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566257 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566312 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566511 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566650 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566803 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.566836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.566826 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.567025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.567000 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.568370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.568347 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.568653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.568635 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.579435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.579408 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.579520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.579446 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvr8\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8\") pod \"maas-default-gateway-openshift-default-6886b96f9-jqhtb\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.614479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.614456 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:02.896607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.896573 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb"] Oct 31 23:08:02.898829 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:08:02.898803 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d96328a_6b55_4673_9ad7_480e8b36dc39.slice/crio-b649ccf1b6364a92e490f51682abee097ec988b07e7bea1c56c9dd629db641e9 WatchSource:0}: Error finding container b649ccf1b6364a92e490f51682abee097ec988b07e7bea1c56c9dd629db641e9: Status 404 returned error can't find the container with id b649ccf1b6364a92e490f51682abee097ec988b07e7bea1c56c9dd629db641e9 Oct 31 23:08:02.900932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.900904 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:02.900995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.900964 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:02.901031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:02.900993 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:03.265072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:03.265039 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" event={"ID":"1d96328a-6b55-4673-9ad7-480e8b36dc39","Type":"ContainerStarted","Data":"74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b"} Oct 31 23:08:03.265230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:03.265102 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" event={"ID":"1d96328a-6b55-4673-9ad7-480e8b36dc39","Type":"ContainerStarted","Data":"b649ccf1b6364a92e490f51682abee097ec988b07e7bea1c56c9dd629db641e9"} Oct 31 23:08:03.294628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:03.294411 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podStartSLOduration=1.2943842380000001 podStartE2EDuration="1.294384238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:03.29048891 +0000 UTC m=+1109.520043322" watchObservedRunningTime="2026-10-31 23:08:03.294384238 +0000 UTC m=+1109.523938643" Oct 31 23:08:03.628636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:03.621877 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:03.638719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:03.638682 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:03.638920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:03.638743 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:04.617891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:04.617854 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:04.618477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:04.618446 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:05.625572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:05.614975 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:05.625572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:05.615051 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:06.622567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:06.614931 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:06.622567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:06.615006 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:07.635497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:07.635461 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:07.636069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:07.636041 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:08.615847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:08.615810 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:08.616023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:08.615865 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:09.615875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:09.615832 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 31 23:08:09.616366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:09.615899 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 31 23:08:10.455574 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:08:10.455515 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:08:10.825573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:10.825526 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:10.826449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:10.826424 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:10.827104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:10.827088 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:08:11.062137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:11.062107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:11.572934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:11.572887 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:13.257947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:13.257915 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:13.402282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:13.402249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:14.809973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:14.809892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:15.018128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:15.018095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:17.317281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.317249 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np"] Oct 31 23:08:17.321270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.321247 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.332368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.332343 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np"] Oct 31 23:08:17.400366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400335 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400375 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400396 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400445 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400517 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400609 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvfn\" (UniqueName: \"kubernetes.io/projected/c08d1d39-5bdc-4834-a0de-af51ef202a20-kube-api-access-zpvfn\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400635 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d1d39-5bdc-4834-a0de-af51ef202a20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400699 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.400766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.400725 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.501878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.501845 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvfn\" (UniqueName: \"kubernetes.io/projected/c08d1d39-5bdc-4834-a0de-af51ef202a20-kube-api-access-zpvfn\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.501885 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d1d39-5bdc-4834-a0de-af51ef202a20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.501908 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.501931 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.501969 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.501997 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502020 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502046 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502254 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502469 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502531 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502616 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.502702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.502683 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08d1d39-5bdc-4834-a0de-af51ef202a20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.504266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.504236 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.504451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.504432 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.516483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.516461 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08d1d39-5bdc-4834-a0de-af51ef202a20-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.516735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.516717 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvfn\" (UniqueName: \"kubernetes.io/projected/c08d1d39-5bdc-4834-a0de-af51ef202a20-kube-api-access-zpvfn\") pod \"maas-default-gateway-openshift-default-6886b96f9-fw9np\" (UID: \"c08d1d39-5bdc-4834-a0de-af51ef202a20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.636749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.636663 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:17.920471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.920448 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np"] Oct 31 23:08:17.922361 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:08:17.922335 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08d1d39_5bdc_4834_a0de_af51ef202a20.slice/crio-cb14e2401475684b4958848675558d1ef1374bbbaa924447b1055ec7d5f4c0c6 WatchSource:0}: Error finding container cb14e2401475684b4958848675558d1ef1374bbbaa924447b1055ec7d5f4c0c6: Status 404 returned error can't find the container with id cb14e2401475684b4958848675558d1ef1374bbbaa924447b1055ec7d5f4c0c6 Oct 31 23:08:17.924518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.924489 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:17.924615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.924589 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:17.924667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:17.924632 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:18.344767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:18.344729 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" event={"ID":"c08d1d39-5bdc-4834-a0de-af51ef202a20","Type":"ContainerStarted","Data":"a36836fa3553fcf52ee5c01c2c365c457826b12ceffd14515fcb72f5a822e9ec"} Oct 31 23:08:18.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:18.344785 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" event={"ID":"c08d1d39-5bdc-4834-a0de-af51ef202a20","Type":"ContainerStarted","Data":"cb14e2401475684b4958848675558d1ef1374bbbaa924447b1055ec7d5f4c0c6"} Oct 31 23:08:18.374209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:18.374157 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podStartSLOduration=1.374145234 podStartE2EDuration="1.374145234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:18.37128633 +0000 UTC m=+1124.600840763" watchObservedRunningTime="2026-10-31 23:08:18.374145234 +0000 UTC m=+1124.603699642" Oct 31 23:08:18.638836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:18.638735 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:18.640586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:18.640561 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:18.640703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:18.640608 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:19.638986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:19.638945 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:19.639450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:19.639003 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:20.638964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:20.638919 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:20.639168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:20.638984 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:21.638917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:21.638883 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:21.639112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:21.638944 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:22.638958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:22.638923 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:22.639409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:22.638984 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:23.637971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:23.637941 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:23.638129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:23.638004 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:23.828560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:23.828514 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:24.637295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:24.637255 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 31 23:08:24.637467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:24.637319 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 31 23:08:25.808758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:25.808725 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:25.809261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:25.809092 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:25.809951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:25.809929 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" Oct 31 23:08:38.481336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.481299 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:38.624387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.624350 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:38.863902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.863871 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p"] Oct 31 23:08:38.868778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.868752 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.882651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.882627 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-04c5c2-openshift-default-dockercfg-js5sc\"" Oct 31 23:08:38.884981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.884955 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p"] Oct 31 23:08:38.921651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921623 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.921793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921664 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.921793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921683 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.921793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921758 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.921970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921895 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.921970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921923 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.922070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.921998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.922070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.922058 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.922177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.922139 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h68n\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:38.968531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.968497 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp"] Oct 31 23:08:38.972267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.972234 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:38.973132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.973112 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:38.987077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.985474 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp"] Oct 31 23:08:38.987077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:38.985614 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a16dfb-openshift-default-dockercfg-lchgz\"" Oct 31 23:08:39.022822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022787 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.022951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022832 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.022951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022856 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.022951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022902 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.022951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022926 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022957 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.022985 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023031 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023057 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023101 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9pd\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023156 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023194 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023234 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023263 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h68n\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023309 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023352 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023358 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023391 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023194 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.023868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.023831 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.024094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.024073 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.024458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.024432 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.026011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.025989 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.026324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.026304 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.045044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.045017 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h68n\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.047084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.047062 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.057814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.057304 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc"] Oct 31 23:08:39.062669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.062650 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.080684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.080661 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc"] Oct 31 23:08:39.081032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.081015 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2294a7-openshift-default-dockercfg-nh2x9\"" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127539 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127608 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127661 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127701 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127752 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127780 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127825 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127849 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127898 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh8c\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127928 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127952 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9pd\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.127975 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128002 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128030 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128073 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128098 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128141 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128177 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.128868 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.129148 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.129353 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.130630 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.130876 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.144170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.133280 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.145575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.144676 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.149463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.149414 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7"] Oct 31 23:08:39.153249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.153163 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9pd\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.153836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.153815 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.154865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.154849 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.176248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.176226 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-255748-openshift-default-dockercfg-89xj2\"" Oct 31 23:08:39.179088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.179067 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7"] Oct 31 23:08:39.182382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.182348 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:39.188343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.188321 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229220 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229279 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh8c\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229313 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229343 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229371 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229410 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229450 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229477 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229517 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229576 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229601 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229670 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcf2\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229706 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229731 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229766 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229778 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.229903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229812 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.230946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.229995 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.230946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.230601 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.230946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.230632 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.230946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.230849 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.231156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.231068 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.231208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.231154 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.232575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.232541 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.233629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.233610 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.248154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.246105 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh8c\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.256115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.256071 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.267957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.267744 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl"] Oct 31 23:08:39.273960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.273921 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.284122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.283410 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-67fce9-openshift-default-dockercfg-vsgs4\"" Oct 31 23:08:39.287989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.287931 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl"] Oct 31 23:08:39.299992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.293312 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332254 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332309 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332346 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcf2\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332375 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332406 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332457 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332479 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332507 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332536 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332580 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332656 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332711 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332737 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332759 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflvl\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332796 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332822 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332845 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.332872 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.333806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.333597 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.336345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.335372 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.336345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.335935 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.340681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.336812 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.340681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.340501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:39.343955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.343636 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.346484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.345968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.346484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.346446 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcf2\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.346661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.346513 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.347587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.347500 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-497d7\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.355123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.352973 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls"] Oct 31 23:08:39.358805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.358790 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.393072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.379582 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls"] Oct 31 23:08:39.393072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.382162 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-135248-openshift-default-dockercfg-g78k6\"" Oct 31 23:08:39.415514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.413789 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.433980 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434031 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434064 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434106 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434133 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434163 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434195 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434223 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflvl\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434265 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434287 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434341 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434378 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434406 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434438 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434463 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434516 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434561 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.434838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.434598 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv78v\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.435736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.435329 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.437788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.435893 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.437788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.436029 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.437788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.436185 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.437788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.436587 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.441353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.441027 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.444289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.444271 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.450289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.450248 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.455968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.454147 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflvl\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.472126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.471907 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:39.509586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.509370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:39.510065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.509709 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p"] Oct 31 23:08:39.514069 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:08:39.514039 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084707a6_e9ab_4b8f_9f76_9f3a8f7d53c2.slice/crio-c4ca6bc0d850fc04bbcba1a196243579fc90d5c58ef7776e71aabe90b7ebee9e WatchSource:0}: Error finding container c4ca6bc0d850fc04bbcba1a196243579fc90d5c58ef7776e71aabe90b7ebee9e: Status 404 returned error can't find the container with id c4ca6bc0d850fc04bbcba1a196243579fc90d5c58ef7776e71aabe90b7ebee9e Oct 31 23:08:39.517630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.517228 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.517630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.517286 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.517630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.517312 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536669 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536710 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536744 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536779 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv78v\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536805 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536824 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536859 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536877 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.536900 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.537333 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.537574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.537526 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.538500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.537764 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.538626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.538591 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.539912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.539886 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.543006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.542980 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.543452 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.543407 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.553732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.553686 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.556979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.556926 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv78v\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.606239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.606195 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:39.642102 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:08:39.636265 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651744fd_77a3_431f_8f0e_a2f8509203f5.slice/crio-138c4da78ba86962a0e250d129360ec56e257547965e459f5907a4cc233f67f4 WatchSource:0}: Error finding container 138c4da78ba86962a0e250d129360ec56e257547965e459f5907a4cc233f67f4: Status 404 returned error can't find the container with id 138c4da78ba86962a0e250d129360ec56e257547965e459f5907a4cc233f67f4 Oct 31 23:08:39.642102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.639192 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.642102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.639295 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.642102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.639338 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.642102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.640990 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp"] Oct 31 23:08:39.648832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.648713 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:39.686389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.686048 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:39.731793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.731760 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc"] Oct 31 23:08:39.737122 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:08:39.737087 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b4dacb_d2b9_4af2_8a5c_cd731ef023a4.slice/crio-fc44bf1bb657a7dd835299d22f9600066ee7502f96ef43f3619a64c59e8d3c53 WatchSource:0}: Error finding container fc44bf1bb657a7dd835299d22f9600066ee7502f96ef43f3619a64c59e8d3c53: Status 404 returned error can't find the container with id fc44bf1bb657a7dd835299d22f9600066ee7502f96ef43f3619a64c59e8d3c53 Oct 31 23:08:39.740702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.740666 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.740823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.740750 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.740823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.740793 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.794715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.794252 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:39.804013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.803978 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7"] Oct 31 23:08:39.824261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.824221 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.824473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.824453 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:39.824628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:39.824607 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.000571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.000497 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl"] Oct 31 23:08:40.011463 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:08:40.011430 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bb011d_a74e_4d78_8a04_79f7eb1dff8f.slice/crio-5e79ec67ea01f6921b29da87cab3e0fc5bfca294d2c51fecdbb9755cefe249a6 WatchSource:0}: Error finding container 5e79ec67ea01f6921b29da87cab3e0fc5bfca294d2c51fecdbb9755cefe249a6: Status 404 returned error can't find the container with id 5e79ec67ea01f6921b29da87cab3e0fc5bfca294d2c51fecdbb9755cefe249a6 Oct 31 23:08:40.013858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.013828 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.014028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.014011 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.014144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.014130 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.046915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.046088 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.046915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.046164 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.046915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.046200 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:08:40.049286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.048588 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls"] Oct 31 23:08:40.058113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.057867 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:40.457319 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:08:40.457265 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:08:40.468609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.468576 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" event={"ID":"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2","Type":"ContainerStarted","Data":"ce420c0a0725ea9b19750cbeca5f764501dee13c2467b916c8bee5365e7fa2cb"} Oct 31 23:08:40.468784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.468638 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" event={"ID":"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2","Type":"ContainerStarted","Data":"c4ca6bc0d850fc04bbcba1a196243579fc90d5c58ef7776e71aabe90b7ebee9e"} Oct 31 23:08:40.470577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.470532 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" event={"ID":"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4","Type":"ContainerStarted","Data":"8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12"} Oct 31 23:08:40.470701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.470590 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" event={"ID":"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4","Type":"ContainerStarted","Data":"fc44bf1bb657a7dd835299d22f9600066ee7502f96ef43f3619a64c59e8d3c53"} Oct 31 23:08:40.472296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.472083 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" event={"ID":"87090fa8-2197-43d2-8bcf-6429c94ef477","Type":"ContainerStarted","Data":"33081f0a5476a3fd7533ac250a7fca332f2b6f2a4a465a7d7c6e145ec761c02c"} Oct 31 23:08:40.472296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.472129 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" event={"ID":"87090fa8-2197-43d2-8bcf-6429c94ef477","Type":"ContainerStarted","Data":"8844c772dce494ff2580c5d3a8a32f57b9c8a4abe72863d268145684f684b819"} Oct 31 23:08:40.473839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.473819 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" event={"ID":"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f","Type":"ContainerStarted","Data":"9bc61faa5e3eb2579d4381e335c0430ff7e623de98b614b5d894a0730177223e"} Oct 31 23:08:40.473946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.473858 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" event={"ID":"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f","Type":"ContainerStarted","Data":"5e79ec67ea01f6921b29da87cab3e0fc5bfca294d2c51fecdbb9755cefe249a6"} Oct 31 23:08:40.475595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.475571 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" event={"ID":"1327691f-6ea3-43b8-8b4b-9e38673bbb03","Type":"ContainerStarted","Data":"4af9753839e3c7534378a9d6f829bcf5f99911cb6daed1b3b97afbdbd759327f"} Oct 31 23:08:40.475672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.475624 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" event={"ID":"1327691f-6ea3-43b8-8b4b-9e38673bbb03","Type":"ContainerStarted","Data":"cdaeac02295a1607313310f89b938aea56a0dbb463ef04692ac587370f23ac5e"} Oct 31 23:08:40.477272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.477251 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" event={"ID":"651744fd-77a3-431f-8f0e-a2f8509203f5","Type":"ContainerStarted","Data":"536d984fedc32cc840d13b1219849d009cb536d3fa27ebff21b72dd1785df755"} Oct 31 23:08:40.477372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.477285 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" event={"ID":"651744fd-77a3-431f-8f0e-a2f8509203f5","Type":"ContainerStarted","Data":"138c4da78ba86962a0e250d129360ec56e257547965e459f5907a4cc233f67f4"} Oct 31 23:08:40.494049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.493998 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podStartSLOduration=2.493986145 podStartE2EDuration="2.493986145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.491720583 +0000 UTC m=+1146.721274995" watchObservedRunningTime="2026-10-31 23:08:40.493986145 +0000 UTC m=+1146.723540555" Oct 31 23:08:40.515905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.515867 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podStartSLOduration=1.515854188 podStartE2EDuration="1.515854188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.514758245 +0000 UTC m=+1146.744312656" watchObservedRunningTime="2026-10-31 23:08:40.515854188 +0000 UTC m=+1146.745408598" Oct 31 23:08:40.540644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.540594 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podStartSLOduration=1.5405812650000001 podStartE2EDuration="1.540581265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.539174542 +0000 UTC m=+1146.768728953" watchObservedRunningTime="2026-10-31 23:08:40.540581265 +0000 UTC m=+1146.770135675" Oct 31 23:08:40.563919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.563863 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podStartSLOduration=1.5638454720000001 podStartE2EDuration="1.563845472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.561695206 +0000 UTC m=+1146.791249617" watchObservedRunningTime="2026-10-31 23:08:40.563845472 +0000 UTC m=+1146.793399883" Oct 31 23:08:40.587742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.587686 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podStartSLOduration=2.587668931 podStartE2EDuration="2.587668931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.583569681 +0000 UTC m=+1146.813124083" watchObservedRunningTime="2026-10-31 23:08:40.587668931 +0000 UTC m=+1146.817223342" Oct 31 23:08:40.607355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.607334 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:40.615168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.615126 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podStartSLOduration=1.615115167 podStartE2EDuration="1.615115167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:40.612904528 +0000 UTC m=+1146.842458938" watchObservedRunningTime="2026-10-31 23:08:40.615115167 +0000 UTC m=+1146.844669576" Oct 31 23:08:40.687247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:40.687214 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:41.188980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.188948 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:41.194091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.194066 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:41.295255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.295224 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:41.301350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.301326 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:41.414298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.414270 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:41.419825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.419788 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:41.472608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.472507 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:41.478802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.478777 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:41.484590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.484537 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:41.486317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.485981 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:41.486317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.486028 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:41.486317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.486076 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:08:41.486317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.486098 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:41.486969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.486951 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:08:41.487068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.487019 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:08:41.487068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.487059 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:08:41.612569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.612518 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:41.692911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:41.692881 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:42.490241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:42.490203 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:42.490850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:42.490824 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:42.491276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:42.491254 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:08:42.491596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:42.491580 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:08:43.014689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:43.014647 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:43.197479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:43.197446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:43.363822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:43.363737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:43.516793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:43.516752 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:43.690707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:43.690628 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:43.835224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:43.835195 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:44.216005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:44.215957 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:44.472353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:44.472269 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:44.964423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:44.964388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:45.113588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:45.110776 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:45.300975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:45.300946 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:45.437333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:45.437302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:45.579093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:45.579019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:45.819779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:45.819747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:45.977674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:45.977585 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:46.132202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:46.132171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:46.297569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:46.285283 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:46.756592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:46.756533 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:46.900100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:46.899986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:47.093060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:47.093028 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:47.225924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:47.225822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:47.363279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:47.363251 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:47.549559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:47.549513 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:47.690725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:47.690676 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:47.911716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:47.910976 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:48.077388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.077350 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:48.230771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.230689 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml"] Oct 31 23:08:48.235930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.235903 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.246707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.246676 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml"] Oct 31 23:08:48.260778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.260728 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:48.345601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.320438 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7d7eed\"" Oct 31 23:08:48.345902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.331782 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7d7eed-dockercfg-4f64k\"" Oct 31 23:08:48.392567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.392117 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.392567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.392169 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdt79\" (UniqueName: \"kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79\") pod \"payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.493608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.493570 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.493785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.493619 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdt79\" (UniqueName: \"kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79\") pod \"payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.494424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.494397 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.511563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.510641 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdt79\" (UniqueName: \"kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79\") pod \"payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.555225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.555173 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:48.902203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.902110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:48.980687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:48.980652 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml"] Oct 31 23:08:49.046221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:49.045923 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:49.183214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:49.183123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:49.549002 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:49.548958 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" event={"ID":"65dd98b9-8631-405f-88bc-f7a4e090177a","Type":"ContainerStarted","Data":"d91d3a59483a95c7a35cedc3406ddb5be23484b379705cfcd397a0d0e0ee6aca"} Oct 31 23:08:49.656875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:49.656835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:50.232571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:50.232307 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:50.380361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:50.380289 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:50.511207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:50.511179 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:50.997202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:50.997160 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:51.142625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:51.142528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:51.279826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:51.279752 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:51.797978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:51.797943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:51.940794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:51.940726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:52.087465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:52.087387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:52.568522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:52.568481 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" event={"ID":"65dd98b9-8631-405f-88bc-f7a4e090177a","Type":"ContainerStarted","Data":"a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe"} Oct 31 23:08:52.568981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:52.568718 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:08:52.613131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:52.613102 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:53.398584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:53.398524 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:54.239084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:54.239047 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:55.001077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:55.001015 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:55.606147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:55.606087 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:56.363431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:56.363396 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:57.149953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:57.149779 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:57.624576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:57.624415 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:58.305607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:58.292129 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:58.431017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:58.430943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:58.548822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:58.548784 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:08:58.549106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:58.549076 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:08:58.710626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:58.709835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:58.902569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:58.877429 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:59.081568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:59.078614 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:59.240225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:59.240017 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:08:59.918571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:08:59.907963 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:01.104570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:01.100490 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:01.665759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:01.662064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:02.368472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:02.367128 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:02.582980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:02.582943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:03.213227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:03.213081 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:03.369050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:03.368977 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:03.506938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:03.506875 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:03.613589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:03.613563 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:09:03.662753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:03.661599 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podStartSLOduration=12.475258799 podStartE2EDuration="15.661529587s" totalImagesPullingTime="3.186270788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:08:52.589822516 +0000 UTC m=+1158.819376926" watchObservedRunningTime="2026-10-31 23:09:03.661529587 +0000 UTC m=+1169.891083997" Oct 31 23:09:03.851418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:03.851239 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:04.005264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:04.005189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:04.159913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:04.159747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:04.583860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:04.583756 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:05.381490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:05.381423 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:06.099081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:06.099037 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:06.904424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:06.904386 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:07.596802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:07.596761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:08.324064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:08.323996 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:08.462976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:08.462942 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:08.608071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:08.607983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:09.004996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:09.004964 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:09.159542 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:09.159507 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:09.326999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:09.326493 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:09.469176 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:09.857889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:10.016262 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:09:10.455617 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:10.820640 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.195112 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4"] Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.266899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.298502 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4"] Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.298608 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.306280 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"openshift-service-ca.crt\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.307799 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"kube-root-ca.crt\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.307886 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"default-dockercfg-wzjmz\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.307951 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"simulator-w0-a16dfb-kserve-self-signed-certs\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.339369 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.339430 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.339566 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lq2\" (UniqueName: \"kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.339621 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.339686 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.339750 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.438315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.441278 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.441366 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.441425 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lq2\" (UniqueName: \"kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.441469 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.441517 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.441651 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.442215 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.443408 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.443714 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.463465 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.470025 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lq2\" (UniqueName: \"kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.471160 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs\") pod \"simulator-w0-a16dfb-kserve-748f84879-rmvq4\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:11.671498 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.029453 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4"] Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:09:12.043659 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6407e3cb_e7ef_48fa_b7fb_320aaff5408e.slice/crio-30a6d0f7ea7404f3e34a98ebd9e0a5f2c5f9b9da769fa2c9d41ba967a5075856 WatchSource:0}: Error finding container 30a6d0f7ea7404f3e34a98ebd9e0a5f2c5f9b9da769fa2c9d41ba967a5075856: Status 404 returned error can't find the container with id 30a6d0f7ea7404f3e34a98ebd9e0a5f2c5f9b9da769fa2c9d41ba967a5075856 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.206198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.206527 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" event={"ID":"6407e3cb-e7ef-48fa-b7fb-320aaff5408e","Type":"ContainerStarted","Data":"16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.206585 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" event={"ID":"6407e3cb-e7ef-48fa-b7fb-320aaff5408e","Type":"ContainerStarted","Data":"30a6d0f7ea7404f3e34a98ebd9e0a5f2c5f9b9da769fa2c9d41ba967a5075856"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.208285 5043 generic.go:356] "Generic (PLEG): container finished" podID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerID="9bc61faa5e3eb2579d4381e335c0430ff7e623de98b614b5d894a0730177223e" exitCode=0 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.216573 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" event={"ID":"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f","Type":"ContainerDied","Data":"9bc61faa5e3eb2579d4381e335c0430ff7e623de98b614b5d894a0730177223e"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.217664 5043 scope.go:122] "RemoveContainer" containerID="9bc61faa5e3eb2579d4381e335c0430ff7e623de98b614b5d894a0730177223e" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.221487 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.221646 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.221696 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.490855 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.491716 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.491786 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.526677 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.526739 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.526884 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.526924 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.527652 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.527683 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.527652 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.527742 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:12.812789 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.176047 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj"] Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.219684 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj"] Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.219813 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.237381 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"openshift-service-ca.crt\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.243438 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"default-dockercfg-jcdwr\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.243606 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"simulator-w1-04c5c2-kserve-self-signed-certs\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.243709 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"kube-root-ca.crt\"" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.263355 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.263404 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.263487 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.263631 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.263690 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.263742 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djd8\" (UniqueName: \"kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.278199 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" event={"ID":"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f","Type":"ContainerStarted","Data":"1e3c5dd51bbb61614ccbdce07be2fbcbd5083fc2914e57121c3ec8733122feac"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.278296 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.279985 5043 generic.go:356] "Generic (PLEG): container finished" podID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerID="4af9753839e3c7534378a9d6f829bcf5f99911cb6daed1b3b97afbdbd759327f" exitCode=0 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.280048 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" event={"ID":"1327691f-6ea3-43b8-8b4b-9e38673bbb03","Type":"ContainerDied","Data":"4af9753839e3c7534378a9d6f829bcf5f99911cb6daed1b3b97afbdbd759327f"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.280391 5043 scope.go:122] "RemoveContainer" containerID="4af9753839e3c7534378a9d6f829bcf5f99911cb6daed1b3b97afbdbd759327f" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.282244 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.282329 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.282375 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.365109 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.365221 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.365259 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.365372 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.365446 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.365513 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djd8\" (UniqueName: \"kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.366342 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.366382 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.366451 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.367582 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.367649 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.376348 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podStartSLOduration=2.3763364080000002 podStartE2EDuration="2.376336408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:13.347810903 +0000 UTC m=+1179.577365314" watchObservedRunningTime="2026-10-31 23:09:13.376336408 +0000 UTC m=+1179.605890817" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.380082 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djd8\" (UniqueName: \"kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8\") pod \"simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.538055 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:13.584857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.577884 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:13.701869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.607688 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:09:13.701869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.608955 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:13.701869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.609013 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:13.701869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.674651 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:13.969875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:13.948373 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj"] Oct 31 23:09:13.969875 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:09:13.951273 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0f8db39_ca3d_4e82_a6de_774b40cb6325.slice/crio-494c1a00eb28d85ae2d0c285ca5efce05149430e767384b4c53f910e1b81ca15 WatchSource:0}: Error finding container 494c1a00eb28d85ae2d0c285ca5efce05149430e767384b4c53f910e1b81ca15: Status 404 returned error can't find the container with id 494c1a00eb28d85ae2d0c285ca5efce05149430e767384b4c53f910e1b81ca15 Oct 31 23:09:14.200798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.200613 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:14.288105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.288068 5043 generic.go:356] "Generic (PLEG): container finished" podID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerID="8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12" exitCode=0 Oct 31 23:09:14.288272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.288138 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" event={"ID":"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4","Type":"ContainerDied","Data":"8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12"} Oct 31 23:09:14.288565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.288519 5043 scope.go:122] "RemoveContainer" containerID="8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12" Oct 31 23:09:14.289750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.289730 5043 generic.go:356] "Generic (PLEG): container finished" podID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerID="33081f0a5476a3fd7533ac250a7fca332f2b6f2a4a465a7d7c6e145ec761c02c" exitCode=0 Oct 31 23:09:14.289854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.289757 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" event={"ID":"87090fa8-2197-43d2-8bcf-6429c94ef477","Type":"ContainerDied","Data":"33081f0a5476a3fd7533ac250a7fca332f2b6f2a4a465a7d7c6e145ec761c02c"} Oct 31 23:09:14.289919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.289856 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.289975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.289918 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.289975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.289951 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.290675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.290332 5043 scope.go:122] "RemoveContainer" containerID="33081f0a5476a3fd7533ac250a7fca332f2b6f2a4a465a7d7c6e145ec761c02c" Oct 31 23:09:14.291645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.291612 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.291736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.291689 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.291736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.291720 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.298823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.298796 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" event={"ID":"1327691f-6ea3-43b8-8b4b-9e38673bbb03","Type":"ContainerStarted","Data":"a0adc13a333dfdbba56b1a5e29430c78284ab010e8f20e3ad316f69d16f26947"} Oct 31 23:09:14.300494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.300473 5043 generic.go:356] "Generic (PLEG): container finished" podID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerID="536d984fedc32cc840d13b1219849d009cb536d3fa27ebff21b72dd1785df755" exitCode=0 Oct 31 23:09:14.300606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.300556 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" event={"ID":"651744fd-77a3-431f-8f0e-a2f8509203f5","Type":"ContainerDied","Data":"536d984fedc32cc840d13b1219849d009cb536d3fa27ebff21b72dd1785df755"} Oct 31 23:09:14.300960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.300941 5043 scope.go:122] "RemoveContainer" containerID="536d984fedc32cc840d13b1219849d009cb536d3fa27ebff21b72dd1785df755" Oct 31 23:09:14.302258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.302239 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" event={"ID":"a0f8db39-ca3d-4e82-a6de-774b40cb6325","Type":"ContainerStarted","Data":"18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd"} Oct 31 23:09:14.302337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.302270 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" event={"ID":"a0f8db39-ca3d-4e82-a6de-774b40cb6325","Type":"ContainerStarted","Data":"494c1a00eb28d85ae2d0c285ca5efce05149430e767384b4c53f910e1b81ca15"} Oct 31 23:09:14.303675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.303658 5043 generic.go:356] "Generic (PLEG): container finished" podID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerID="ce420c0a0725ea9b19750cbeca5f764501dee13c2467b916c8bee5365e7fa2cb" exitCode=0 Oct 31 23:09:14.303751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.303736 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" event={"ID":"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2","Type":"ContainerDied","Data":"ce420c0a0725ea9b19750cbeca5f764501dee13c2467b916c8bee5365e7fa2cb"} Oct 31 23:09:14.304047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.304030 5043 scope.go:122] "RemoveContainer" containerID="ce420c0a0725ea9b19750cbeca5f764501dee13c2467b916c8bee5365e7fa2cb" Oct 31 23:09:14.306860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.306833 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.306935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.306891 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.306935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.306922 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.307005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.306945 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.307039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.307013 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.307082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.307063 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:09:14.509144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.509106 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podStartSLOduration=1.5090900569999999 podStartE2EDuration="1.509090057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:14.507145384 +0000 UTC m=+1180.736699795" watchObservedRunningTime="2026-10-31 23:09:14.509090057 +0000 UTC m=+1180.738644465" Oct 31 23:09:14.592781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.589906 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:14.635570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.621097 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:14.635570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.621149 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:14.701236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.701009 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:09:14.701236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.701131 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:14.701236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.701172 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:14.906399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:14.906298 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:15.328569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.328434 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" event={"ID":"651744fd-77a3-431f-8f0e-a2f8509203f5","Type":"ContainerStarted","Data":"e29ef966fe50e855687948706d2a78d30d1f4e59383978322d4d6003ad55b408"} Oct 31 23:09:15.352568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.350462 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" event={"ID":"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2","Type":"ContainerStarted","Data":"b814ef7c3ca182666c0e2e8b2632d1ef0cfbdbba1c04f65825d7c0d0dad25522"} Oct 31 23:09:15.380689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.379928 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" event={"ID":"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4","Type":"ContainerStarted","Data":"ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1"} Oct 31 23:09:15.392868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.391351 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" event={"ID":"87090fa8-2197-43d2-8bcf-6429c94ef477","Type":"ContainerStarted","Data":"cf7e673bb289d5670fabaa0b72aa93934c4ebf59e72e401a901ca53b45194996"} Oct 31 23:09:15.416988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.416959 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:09:15.417353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.417320 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:15.417507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.417487 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:15.481565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.478448 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:09:15.481565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.478766 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:15.481565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.478816 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:15.519640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.519554 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:15.626691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.626591 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:15.627237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.627203 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:15.723573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.723019 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:15.723573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:15.723083 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:16.195389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.194985 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:09:16.195389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.195108 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:16.195389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.195150 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:16.313036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.313000 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:16.313799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.313777 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:09:16.314040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.314022 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:16.314171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.314151 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:16.429209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.429177 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:16.429489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.429463 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:16.463269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.463172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:16.474416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.474385 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:16.474901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.474861 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:16.611139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.610716 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:16.611139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.610773 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:16.687572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.686967 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:16.687572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:16.687030 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:17.032065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.032028 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:17.189028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.188989 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:17.189308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.189280 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:17.297025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.296936 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:17.297259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.297230 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:17.426659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.426621 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:17.449528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.449492 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:17.473518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.472410 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:17.473518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.472462 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:17.610135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.610037 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:17.610404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.610363 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:17.687036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.687000 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:17.687342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.687296 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:17.706110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:17.706070 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:18.189944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.189906 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:18.191066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.191032 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:18.317844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.317804 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:18.318131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.318101 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:18.324022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.323991 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:18.422959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.422923 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:18.423975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.423939 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:18.487627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.484102 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:18.487627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.484161 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:18.607569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.607002 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:18.607569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.607062 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:18.703602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.688910 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:18.703602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:18.688974 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:19.112447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.112268 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:19.192373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.191938 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:19.192373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.191994 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:19.267200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.267162 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn"] Oct 31 23:09:19.278434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.278402 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.304585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.300672 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:19.304585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.300728 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:19.307727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.306190 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn"] Oct 31 23:09:19.307727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.306473 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"kube-root-ca.crt\"" Oct 31 23:09:19.307727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.307182 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"simulator-w5-2294a7-kserve-self-signed-certs\"" Oct 31 23:09:19.307727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.307273 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"default-dockercfg-l9tlv\"" Oct 31 23:09:19.307727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.307332 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"openshift-service-ca.crt\"" Oct 31 23:09:19.374236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.373999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:19.416579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.416146 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.416579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.416238 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.416579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.416282 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.416579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.416414 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.416579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.416474 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzf5\" (UniqueName: \"kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.416579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.416504 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.417559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.417521 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:19.423566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.419635 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:19.473125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.473092 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:19.473393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.473366 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:19.524491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.524454 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.524826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.524805 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzf5\" (UniqueName: \"kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.524961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.524946 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.525082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.525069 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.525246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.525232 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.525369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.525357 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.526119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.526101 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.526943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.526912 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.527702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.527684 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.671487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.671402 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzf5\" (UniqueName: \"kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.720672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.672005 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.721676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.672499 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs\") pod \"simulator-w5-2294a7-kserve-58c5569cf9-mrgbn\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:19.721849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.672632 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:19.722020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.721979 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:19.722525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.712045 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:19.722730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.722708 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:19.906273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.906228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:19.987617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:19.986713 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:20.225571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.196576 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:20.226388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.226343 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:20.455569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.455292 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:20.455569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.455358 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:20.469134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.469096 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:20.469394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.469365 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:20.485495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.485388 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:20.485495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.485441 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:20.569574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.559102 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:20.629620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.626816 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 23:09:20.629620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.626873 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 23:09:20.692122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.692046 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:20.698742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.698617 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:20.910949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:20.900060 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn"] Oct 31 23:09:21.058748 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:09:21.057265 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a0ded48_b028_4b3b_99ac_665ba58e14f6.slice/crio-3a0307ba1cc6d4d6776ab31dc6519c3762d505471cad4f5a90c21b6e126c5c84\": RecentStats: unable to find data in memory cache]" Oct 31 23:09:21.192740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.192655 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:21.192985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.192953 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:21.298506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.298469 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:21.299107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.299071 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:21.349571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.340994 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:21.425568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.416255 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:21.425568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.416314 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:21.478113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.477694 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:21.478113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.477751 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:21.655471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.655441 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:09:21.657047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.657025 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:09:21.665014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.664989 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:09:21.675229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.675206 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:21.694660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.694627 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:21.694899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:21.694872 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:22.055567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.055334 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:22.084566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.084444 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" event={"ID":"5a0ded48-b028-4b3b-99ac-665ba58e14f6","Type":"ContainerStarted","Data":"3a0307ba1cc6d4d6776ab31dc6519c3762d505471cad4f5a90c21b6e126c5c84"} Oct 31 23:09:22.191030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.190996 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:22.191321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.191272 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:22.299105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.299074 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:22.299718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.299671 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:22.467016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.466918 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:22.467291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.467247 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:22.469509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.469338 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:22.473929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.473758 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:22.473929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.473798 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:22.482573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.478232 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:22.687050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.687015 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:22.687345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.687299 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:22.708307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:22.708271 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:23.137686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.137626 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" event={"ID":"5a0ded48-b028-4b3b-99ac-665ba58e14f6","Type":"ContainerStarted","Data":"2c9638959b2e675c1e2d903c50f23d0278240b35e1e1e31c7331c16adcef7f9e"} Oct 31 23:09:23.162180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.162122 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podStartSLOduration=4.162102198 podStartE2EDuration="4.162102198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:23.159340106 +0000 UTC m=+1189.388894534" watchObservedRunningTime="2026-10-31 23:09:23.162102198 +0000 UTC m=+1189.391656610" Oct 31 23:09:23.194941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.194905 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:23.195198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.195172 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:23.294988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.294948 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:23.295249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.295219 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:23.314434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.314382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:23.416319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.416226 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:23.416569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.416518 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:23.486083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.486049 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:23.486328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.486304 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:23.528708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.528661 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:23.592764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.592722 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:23.703744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.703647 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:09:23.703744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:23.703711 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:09:24.021367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.021323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:24.104393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.104362 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:24.192569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.191959 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:24.192569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.192037 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:24.233586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.225123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:24.298038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.297949 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:24.298280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.298252 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:24.447596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.447541 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:24.448156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.448124 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:24.483220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.483185 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 23:09:24.483462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.483431 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 23:09:24.529176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.529046 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:24.600850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.600751 5043 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:09:24.749083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.749045 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:24.871711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.871624 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:09:24.900193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.900164 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:09:24.911842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:24.911809 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:09:25.090060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.090026 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:09:25.192950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.192865 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 31 23:09:25.193189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.193164 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 31 23:09:25.297911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.295270 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:09:25.297911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.295340 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:09:25.416909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.415596 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 31 23:09:25.416909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.415676 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 31 23:09:25.434119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.434059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:25.602713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.602666 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:09:25.604963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.604941 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:09:25.605953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:25.605938 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:09:26.091044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.090994 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:26.214855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.214827 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:09:26.216145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.216124 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:09:26.223562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.221706 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:09:26.318567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.318179 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:09:26.323564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.319749 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:09:26.334015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.333989 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:09:26.419402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.419326 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:09:26.419926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.419899 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:09:26.420627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.420605 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:09:26.735360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.735249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:26.881998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:26.881959 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:27.489358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:27.484309 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:28.102995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:28.102959 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:28.842701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:28.842650 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:29.702676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:29.702644 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:29.988050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:29.987971 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:30.000781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:30.000751 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:30.289041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:30.289010 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:09:30.414917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:30.414882 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:31.248365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:31.248304 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:31.925947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:31.925910 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:32.723808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:32.723771 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:33.317699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:33.317665 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:33.595461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:33.595381 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:33.609075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:33.609048 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:09:34.007885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:34.007852 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:34.072408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:34.072378 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:09:34.077848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:34.077821 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:09:34.621713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:34.621677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:35.403792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:35.403757 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:35.535329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:35.535296 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:36.027067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:36.026999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:36.910182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:36.910137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:37.695261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:37.695227 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:38.511688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.511654 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:38.677100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.677057 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z"] Oct 31 23:09:38.682453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.682431 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.691686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.691661 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z"] Oct 31 23:09:38.692910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.692882 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"premium-w3-67fce9-kserve-self-signed-certs\"" Oct 31 23:09:38.693013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.692888 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"openshift-service-ca.crt\"" Oct 31 23:09:38.693838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.693509 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"default-dockercfg-7cn9l\"" Oct 31 23:09:38.694114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.694089 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"kube-root-ca.crt\"" Oct 31 23:09:38.720029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.720003 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.720159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.720052 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.720159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.720067 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2v8n\" (UniqueName: \"kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.720231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.720188 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.720264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.720251 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.720294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.720284 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.758235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.758207 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:38.822849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.822738 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.823675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.823644 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.824652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.823847 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.824876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.824856 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.825023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.825008 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.825137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.825120 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2v8n\" (UniqueName: \"kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.825314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.824940 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.826949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.825587 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.827369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.827347 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.859178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.859136 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.861127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.861096 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:38.861924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:38.861900 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2v8n\" (UniqueName: \"kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n\") pod \"premium-w3-67fce9-kserve-559687cd89-bqv4z\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:39.009353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:39.009318 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:39.195455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:39.195382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:39.293572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:39.293537 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z"] Oct 31 23:09:39.295788 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:09:39.295748 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d2bc5af_64a6_4051_8f0a_ed890a14a7f5.slice/crio-00d2023a6256168218bfc3eff55824130dd642fdb698f24a3a0f6d17fe2dfd68 WatchSource:0}: Error finding container 00d2023a6256168218bfc3eff55824130dd642fdb698f24a3a0f6d17fe2dfd68: Status 404 returned error can't find the container with id 00d2023a6256168218bfc3eff55824130dd642fdb698f24a3a0f6d17fe2dfd68 Oct 31 23:09:39.355650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:39.355605 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" event={"ID":"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5","Type":"ContainerStarted","Data":"5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619"} Oct 31 23:09:39.355743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:39.355650 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" event={"ID":"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5","Type":"ContainerStarted","Data":"00d2023a6256168218bfc3eff55824130dd642fdb698f24a3a0f6d17fe2dfd68"} Oct 31 23:09:39.914695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:39.914646 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:40.384246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:40.384198 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podStartSLOduration=2.384183236 podStartE2EDuration="2.384183236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:40.38143359 +0000 UTC m=+1206.610987993" watchObservedRunningTime="2026-10-31 23:09:40.384183236 +0000 UTC m=+1206.613737644" Oct 31 23:09:40.455708 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:09:40.455670 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:09:40.497812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:40.497749 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:41.010209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:41.010172 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:41.102322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:41.102292 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:41.747568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:41.745813 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:42.472764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:42.472527 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:42.612441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:42.612411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:43.276311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:43.276274 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:44.033134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.033092 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:44.520591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.514497 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls"] Oct 31 23:09:44.532384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.532354 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.532590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.532565 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls"] Oct 31 23:09:44.572822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.570406 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"premium-w5-2294a7-kserve-self-signed-certs\"" Oct 31 23:09:44.587637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.587608 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:44.612082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.612050 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.612326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.612158 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.612326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.612222 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.612326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.612256 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.612499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.612353 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.612499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.612379 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq778\" (UniqueName: \"kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.713845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.713807 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.713897 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.713930 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.714051 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.714154 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.714194 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq778\" (UniqueName: \"kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.714274 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.714306 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.714425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.714356 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.717578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.716704 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.717578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.717011 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.727314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.727286 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq778\" (UniqueName: \"kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778\") pod \"premium-w5-2294a7-kserve-6d7b7f4f94-8jbls\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:44.812294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.812203 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:44.890283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:44.890248 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:45.173353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:45.173328 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls"] Oct 31 23:09:45.174985 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:09:45.174962 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf23336de_2435_41dd_a86e_629d91536e27.slice/crio-e41ab56848321e54fa68b4fb0fec6d4cffdadffe0788b17b291ad551080af372 WatchSource:0}: Error finding container e41ab56848321e54fa68b4fb0fec6d4cffdadffe0788b17b291ad551080af372: Status 404 returned error can't find the container with id e41ab56848321e54fa68b4fb0fec6d4cffdadffe0788b17b291ad551080af372 Oct 31 23:09:45.397752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:45.397716 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" event={"ID":"f23336de-2435-41dd-a86e-629d91536e27","Type":"ContainerStarted","Data":"a56c04ce6a42b4c3ab48ad4ecb4243c63177ccdfb0d809e8f43ae2151b95cbd1"} Oct 31 23:09:45.397922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:45.397761 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" event={"ID":"f23336de-2435-41dd-a86e-629d91536e27","Type":"ContainerStarted","Data":"e41ab56848321e54fa68b4fb0fec6d4cffdadffe0788b17b291ad551080af372"} Oct 31 23:09:45.418705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:45.418639 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podStartSLOduration=1.418627729 podStartE2EDuration="1.418627729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:45.41808999 +0000 UTC m=+1211.647644401" watchObservedRunningTime="2026-10-31 23:09:45.418627729 +0000 UTC m=+1211.648182173" Oct 31 23:09:45.464275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:45.464206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:45.890474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:45.890437 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:46.201396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:46.201306 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:46.819097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:46.819062 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:47.493165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:47.493121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:48.197297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:48.197235 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:48.908968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:48.908939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:49.015977 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:49.015947 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:49.050874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:49.050843 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:49.064502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:49.064476 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:49.434590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:49.434560 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:09:49.596668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:49.596625 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:50.204910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:50.204880 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:50.919806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:50.919758 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:51.594961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.594923 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:51.649184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.649157 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv"] Oct 31 23:09:51.653942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.653920 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.662701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.662672 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"openshift-service-ca.crt\"" Oct 31 23:09:51.663377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.663361 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"kube-root-ca.crt\"" Oct 31 23:09:51.663533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.663367 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"default-dockercfg-lql6f\"" Oct 31 23:09:51.663762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.663745 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7d7eed\"/\"premium-w6-7d7eed-kserve-self-signed-certs\"" Oct 31 23:09:51.665592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.665537 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv"] Oct 31 23:09:51.720192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.720162 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.720323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.720219 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.720369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.720341 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.720415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.720402 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.720577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.720560 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.720634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.720596 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.741009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.740980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.821958 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.822007 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.822087 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.822137 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.822243 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.822310 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.822767 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.823045 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.823707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.823305 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.825422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.825350 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.828916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.828893 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.836899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.836876 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx\") pod \"premium-w6-7d7eed-kserve-5599dcf654-vx4rv\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:51.968310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:51.968235 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:52.249079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:52.249047 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv"] Oct 31 23:09:52.250140 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:09:52.250073 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod160baa62_7a99_44ef_ac94_58f851fbc7a5.slice/crio-44004b0a404a7c8cd1cf722fcaf0578bf4c2b385dae2d0a8647172e95eb13a27 WatchSource:0}: Error finding container 44004b0a404a7c8cd1cf722fcaf0578bf4c2b385dae2d0a8647172e95eb13a27: Status 404 returned error can't find the container with id 44004b0a404a7c8cd1cf722fcaf0578bf4c2b385dae2d0a8647172e95eb13a27 Oct 31 23:09:52.286307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:52.286284 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:52.444631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:52.444590 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" event={"ID":"160baa62-7a99-44ef-ac94-58f851fbc7a5","Type":"ContainerStarted","Data":"264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9"} Oct 31 23:09:52.444810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:52.444645 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" event={"ID":"160baa62-7a99-44ef-ac94-58f851fbc7a5","Type":"ContainerStarted","Data":"44004b0a404a7c8cd1cf722fcaf0578bf4c2b385dae2d0a8647172e95eb13a27"} Oct 31 23:09:52.468912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:52.468867 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podStartSLOduration=1.468854242 podStartE2EDuration="1.468854242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:09:52.466980983 +0000 UTC m=+1218.696535395" watchObservedRunningTime="2026-10-31 23:09:52.468854242 +0000 UTC m=+1218.698408651" Oct 31 23:09:52.968994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:52.968960 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:09:53.005673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:53.005645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:53.709462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:53.709431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:54.651644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:54.651568 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:54.893690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:54.893624 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:54.912942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:54.912855 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:55.588472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:55.588441 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:09:56.489980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:56.489941 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:56.828232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:56.828198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:57.421096 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:57.421051 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:58.116795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:58.116761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:58.865128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:58.865090 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:09:59.561434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:09:59.561397 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:00.396864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:00.396821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:01.031197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:01.031154 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:01.696834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:01.696779 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:01.969300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:01.969216 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:10:01.981168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:01.981146 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:10:02.393836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:02.393798 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:02.566868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:02.566803 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:10:03.207168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:03.207134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:04.025316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:04.025260 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:04.712445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:04.712398 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:05.511577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:05.511446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:06.112336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:06.112302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:06.734080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:06.733848 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:07.320491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:07.320451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:08.038427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:08.038384 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:09.157540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:09.156213 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:09.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:09.944741 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:10.460318 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:10:10.460274 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:10:10.824702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:10.824662 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:11.636895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:11.636857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:12.358943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:12.358907 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:13.007123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:13.007085 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:13.634124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:13.634055 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:14.308939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:14.308813 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:15.310812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:15.310297 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:15.911575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:15.911510 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:16.505950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:16.505909 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:17.300976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:17.300943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:17.918503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:17.918472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:18.493236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:18.493198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:19.107422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:19.107391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:19.878204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:19.878165 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:20.508155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:20.508101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:21.211684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:21.211642 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:21.823528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:21.823492 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:22.101657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.101587 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx"] Oct 31 23:10:22.105293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.105274 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.111183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.111165 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:10:22.111413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.111394 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:10:22.111526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.111489 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:10:22.111618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.111605 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:10:22.113057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.113031 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx"] Oct 31 23:10:22.146459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.146425 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.146648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.146592 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.146648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.146637 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkp72\" (UniqueName: \"kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.247708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.247672 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.247874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.247717 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkp72\" (UniqueName: \"kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.247874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.247815 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.248483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.248456 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.250125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.250093 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.260873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.260849 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkp72\" (UniqueName: \"kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72\") pod \"maas-api-revoke-sub-532b7399-zqpmx\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.422540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.422467 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:22.576026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.575992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:22.703874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.703843 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx"] Oct 31 23:10:22.705809 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:22.705782 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace8520f_c240_4ab4_ae90_20a1be000980.slice/crio-e37ba5b2ac38a619c1ca1beb05a870cb9c6f756da27afbac48cf42bed4394d1f WatchSource:0}: Error finding container e37ba5b2ac38a619c1ca1beb05a870cb9c6f756da27afbac48cf42bed4394d1f: Status 404 returned error can't find the container with id e37ba5b2ac38a619c1ca1beb05a870cb9c6f756da27afbac48cf42bed4394d1f Oct 31 23:10:22.707873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:22.707855 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:10:23.326164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:23.326126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:23.721602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:23.721493 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" event={"ID":"ace8520f-c240-4ab4-ae90-20a1be000980","Type":"ContainerStarted","Data":"e37ba5b2ac38a619c1ca1beb05a870cb9c6f756da27afbac48cf42bed4394d1f"} Oct 31 23:10:24.013164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.013129 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:24.255018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.254978 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:24.451173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.451010 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:24.635469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.635430 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr"] Oct 31 23:10:24.640639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.640615 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.651411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.651384 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7d7eed-openshift-default-dockercfg-lp7z5\"" Oct 31 23:10:24.664150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.658936 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr"] Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671505 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671609 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671651 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671714 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfc6b\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671798 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671826 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671881 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671904 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.692566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.671932 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.773314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.773262 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.773638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.773611 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.773825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.773808 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfc6b\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.773950 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.773986 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.774024 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.774086 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.774112 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.774146 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.774238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.774574 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.775834 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.776264 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.776698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.776344 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.778187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.778159 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.778944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.778919 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.788716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.788673 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfc6b\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.789144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.789123 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:24.943117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.943039 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:24.956955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:24.956767 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:25.287937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.287319 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr"] Oct 31 23:10:25.292342 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:25.290625 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9e7b2a_02ec_4c6d_bea4_6dc49798d296.slice/crio-e1ce472123a81e50b4a4ce1a51e9ff4c503486813e112eb37f904f172ad8fc11 WatchSource:0}: Error finding container e1ce472123a81e50b4a4ce1a51e9ff4c503486813e112eb37f904f172ad8fc11: Status 404 returned error can't find the container with id e1ce472123a81e50b4a4ce1a51e9ff4c503486813e112eb37f904f172ad8fc11 Oct 31 23:10:25.294564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.294441 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:10:25.294564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.294513 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:10:25.294727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.294582 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:10:25.757369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.756614 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" event={"ID":"be9e7b2a-02ec-4c6d-bea4-6dc49798d296","Type":"ContainerStarted","Data":"ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355"} Oct 31 23:10:25.757369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.756684 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" event={"ID":"be9e7b2a-02ec-4c6d-bea4-6dc49798d296","Type":"ContainerStarted","Data":"e1ce472123a81e50b4a4ce1a51e9ff4c503486813e112eb37f904f172ad8fc11"} Oct 31 23:10:25.771860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.771820 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:25.790646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.786722 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podStartSLOduration=1.786699734 podStartE2EDuration="1.786699734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:25.782677898 +0000 UTC m=+1252.012232307" watchObservedRunningTime="2026-10-31 23:10:25.786699734 +0000 UTC m=+1252.016254142" Oct 31 23:10:25.958597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:25.958511 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:26.746570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:26.737146 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:26.965576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:26.959674 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 23:10:26.965576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:26.959743 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 23:10:27.465925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:27.465892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:27.957565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:27.957531 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 23:10:27.957645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:27.957599 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 23:10:28.183357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:28.183319 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:28.782373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:28.782341 5043 generic.go:356] "Generic (PLEG): container finished" podID="ace8520f-c240-4ab4-ae90-20a1be000980" containerID="ebee9ac71b5803c68c4e4128fb60cd6f16459cc22ae9eb8a19a66075576fbc46" exitCode=0 Oct 31 23:10:28.782565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:28.782402 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" event={"ID":"ace8520f-c240-4ab4-ae90-20a1be000980","Type":"ContainerDied","Data":"ebee9ac71b5803c68c4e4128fb60cd6f16459cc22ae9eb8a19a66075576fbc46"} Oct 31 23:10:28.911019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:28.910919 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:28.961053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:28.961015 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 23:10:28.961319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:28.961291 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 23:10:29.733575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:29.732065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:29.957603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:29.957519 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 23:10:29.957603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:29.957592 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 23:10:29.983104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:29.983077 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:30.072761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.072726 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca\") pod \"ace8520f-c240-4ab4-ae90-20a1be000980\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " Oct 31 23:10:30.072958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.072826 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72\" (UniqueName: \"kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72\") pod \"ace8520f-c240-4ab4-ae90-20a1be000980\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " Oct 31 23:10:30.072958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.072852 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca\") pod \"ace8520f-c240-4ab4-ae90-20a1be000980\" (UID: \"ace8520f-c240-4ab4-ae90-20a1be000980\") " Oct 31 23:10:30.073951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.073919 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca" pod "ace8520f-c240-4ab4-ae90-20a1be000980" (UID: "ace8520f-c240-4ab4-ae90-20a1be000980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:30.075473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.075447 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72" pod "ace8520f-c240-4ab4-ae90-20a1be000980" (UID: "ace8520f-c240-4ab4-ae90-20a1be000980"). InnerVolumeSpecName "kube-api-access-tkp72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:30.076312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.076274 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca" pod "ace8520f-c240-4ab4-ae90-20a1be000980" (UID: "ace8520f-c240-4ab4-ae90-20a1be000980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:30.174007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.173974 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:30.174007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.174005 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkp72\" (UniqueName: \"kubernetes.io/projected/ace8520f-c240-4ab4-ae90-20a1be000980-kube-api-access-tkp72\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:30.174206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.174016 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace8520f-c240-4ab4-ae90-20a1be000980-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:30.435496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.435402 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:30.796106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.796076 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" Oct 31 23:10:30.796610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.796122 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx" event={"ID":"ace8520f-c240-4ab4-ae90-20a1be000980","Type":"ContainerDied","Data":"e37ba5b2ac38a619c1ca1beb05a870cb9c6f756da27afbac48cf42bed4394d1f"} Oct 31 23:10:30.796610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.796173 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37ba5b2ac38a619c1ca1beb05a870cb9c6f756da27afbac48cf42bed4394d1f" Oct 31 23:10:30.958062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.958024 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 23:10:30.958238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:30.958092 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 23:10:31.207635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:31.207528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:31.860876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:31.860839 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:32.034097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:32.034051 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:32.034592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:32.034568 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:32.036810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:32.036789 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:10:32.609815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:32.609787 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:33.249736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:33.249707 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:33.898574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:33.898521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:34.518959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:34.518923 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:35.122011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:35.121967 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:35.732090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:35.732055 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:36.340667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:36.340626 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:36.972487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:36.972445 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:37.585600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:37.585531 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:38.388603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:38.388565 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:39.115890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.115857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:39.364444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.364408 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm"] Oct 31 23:10:39.365254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.365225 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ace8520f-c240-4ab4-ae90-20a1be000980" containerName="revoke-keys" Oct 31 23:10:39.365254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.365249 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace8520f-c240-4ab4-ae90-20a1be000980" containerName="revoke-keys" Oct 31 23:10:39.365407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.365395 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace8520f-c240-4ab4-ae90-20a1be000980" containerName="revoke-keys" Oct 31 23:10:39.372680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.371338 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.377227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.377188 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:10:39.378464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.378432 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:10:39.378707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.378685 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:10:39.378923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.378909 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:10:39.382854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.382828 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm"] Oct 31 23:10:39.443047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.443010 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.443811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.443786 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.447052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.447029 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22z5\" (UniqueName: \"kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.548314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.548236 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.548314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.548305 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22z5\" (UniqueName: \"kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.548602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.548462 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.549778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.549749 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.551746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.551718 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.564681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.564650 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22z5\" (UniqueName: \"kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5\") pod \"maas-api-revoke-sub-720e97ed-fvxcm\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.698023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.697941 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:39.814201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:39.810275 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:40.358586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.353707 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm"] Oct 31 23:10:40.468309 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:10:40.468272 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:10:40.482090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.482056 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt"] Oct 31 23:10:40.488893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.488865 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.497221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.497182 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt"] Oct 31 23:10:40.599460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.599388 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.599660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.599569 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.599716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.599693 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwv5\" (UniqueName: \"kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.614570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.612255 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:40.702768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.702139 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.702768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.702200 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.702768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.702262 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwv5\" (UniqueName: \"kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.703068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.703003 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.719443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.719417 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwv5\" (UniqueName: \"kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.740166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.738241 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-881f57fe-g4xwt\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.839796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.836744 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:40.946952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.946909 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r"] Oct 31 23:10:40.969596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.969514 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" event={"ID":"227acc28-b8eb-4ab9-a764-83b71dc868e8","Type":"ContainerStarted","Data":"82b9edbf43c7962c313cd1f8f5ff4e0082d2323ee296a317fa24fd9ae7d77dda"} Oct 31 23:10:40.969596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.969596 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" event={"ID":"227acc28-b8eb-4ab9-a764-83b71dc868e8","Type":"ContainerStarted","Data":"b57107c09ee22573f4468dcb6f148480e9d74f520b24471f3307a30d3f4e3aa6"} Oct 31 23:10:40.969837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.969689 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:40.974847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:40.974817 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r"] Oct 31 23:10:41.014197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.011515 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" podStartSLOduration=2.011498059 podStartE2EDuration="2.011498059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:40.988713906 +0000 UTC m=+1267.218268318" watchObservedRunningTime="2026-10-31 23:10:41.011498059 +0000 UTC m=+1267.241052476" Oct 31 23:10:41.106053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.105956 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.106053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.106028 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.106255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.106136 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcb8b\" (UniqueName: \"kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.135972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.135933 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt"] Oct 31 23:10:41.140898 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:41.140873 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9578f643_4094_4d1d_be0d_5493f292a27d.slice/crio-17569d9971bc72f4888ad3a6c0a095cd23d369318051a9eab2d4995675c9caf1 WatchSource:0}: Error finding container 17569d9971bc72f4888ad3a6c0a095cd23d369318051a9eab2d4995675c9caf1: Status 404 returned error can't find the container with id 17569d9971bc72f4888ad3a6c0a095cd23d369318051a9eab2d4995675c9caf1 Oct 31 23:10:41.207788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.207755 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.207905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.207803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.207905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.207840 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcb8b\" (UniqueName: \"kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.208619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.208596 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.210084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.210068 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.220407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.220382 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcb8b\" (UniqueName: \"kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b\") pod \"maas-api-revoke-sub-f5890f43-z598r\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.288041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.283413 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:41.288648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.288618 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87"] Oct 31 23:10:41.301566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.296409 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:41.301566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.299208 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.309634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.302569 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87"] Oct 31 23:10:41.410812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.410450 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.410812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.410602 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvckn\" (UniqueName: \"kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.410812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.410635 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.513094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.511830 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.513094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.511991 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvckn\" (UniqueName: \"kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.513094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.512022 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.514482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.514125 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.518969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.518941 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.544448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.544417 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvckn\" (UniqueName: \"kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn\") pod \"maas-api-revoke-sub-a56b98fe-lmv87\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.630647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.630602 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r"] Oct 31 23:10:41.633370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.633344 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:41.633923 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:41.633895 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e60aa67_7463_4fe2_8154_2ca468cbd804.slice/crio-40c65ee7eb8ab2b0ba4b19d5f30004f30de6bbe0eba4f9275fb38c2c142cdbbf WatchSource:0}: Error finding container 40c65ee7eb8ab2b0ba4b19d5f30004f30de6bbe0eba4f9275fb38c2c142cdbbf: Status 404 returned error can't find the container with id 40c65ee7eb8ab2b0ba4b19d5f30004f30de6bbe0eba4f9275fb38c2c142cdbbf Oct 31 23:10:41.951762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.951736 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87"] Oct 31 23:10:41.952960 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:41.952918 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe928e72_e3a3_484c_8b4e_aa642aed31fc.slice/crio-2010e8be7eb011bff895c29e65c3edb56f246582e20b4c63c7c9a5999981d2b0 WatchSource:0}: Error finding container 2010e8be7eb011bff895c29e65c3edb56f246582e20b4c63c7c9a5999981d2b0: Status 404 returned error can't find the container with id 2010e8be7eb011bff895c29e65c3edb56f246582e20b4c63c7c9a5999981d2b0 Oct 31 23:10:41.974771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.974739 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" event={"ID":"9578f643-4094-4d1d-be0d-5493f292a27d","Type":"ContainerStarted","Data":"0dbc2776b0634116d6327fa6d89b011694955e469aa5bfb720307febd416f97f"} Oct 31 23:10:41.974969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.974951 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" event={"ID":"9578f643-4094-4d1d-be0d-5493f292a27d","Type":"ContainerStarted","Data":"17569d9971bc72f4888ad3a6c0a095cd23d369318051a9eab2d4995675c9caf1"} Oct 31 23:10:41.976514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.976487 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" event={"ID":"7e60aa67-7463-4fe2-8154-2ca468cbd804","Type":"ContainerStarted","Data":"40c65ee7eb8ab2b0ba4b19d5f30004f30de6bbe0eba4f9275fb38c2c142cdbbf"} Oct 31 23:10:41.978338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.978314 5043 generic.go:356] "Generic (PLEG): container finished" podID="227acc28-b8eb-4ab9-a764-83b71dc868e8" containerID="82b9edbf43c7962c313cd1f8f5ff4e0082d2323ee296a317fa24fd9ae7d77dda" exitCode=0 Oct 31 23:10:41.978438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.978381 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" event={"ID":"227acc28-b8eb-4ab9-a764-83b71dc868e8","Type":"ContainerDied","Data":"82b9edbf43c7962c313cd1f8f5ff4e0082d2323ee296a317fa24fd9ae7d77dda"} Oct 31 23:10:41.979448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:41.979426 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" event={"ID":"fe928e72-e3a3-484c-8b4e-aa642aed31fc","Type":"ContainerStarted","Data":"2010e8be7eb011bff895c29e65c3edb56f246582e20b4c63c7c9a5999981d2b0"} Oct 31 23:10:42.001178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.001124 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" podStartSLOduration=2.001108199 podStartE2EDuration="2.001108199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:42.000061297 +0000 UTC m=+1268.229615721" watchObservedRunningTime="2026-10-31 23:10:42.001108199 +0000 UTC m=+1268.230662611" Oct 31 23:10:42.009365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.009331 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:42.695646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.695610 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:42.987895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.987810 5043 generic.go:356] "Generic (PLEG): container finished" podID="9578f643-4094-4d1d-be0d-5493f292a27d" containerID="0dbc2776b0634116d6327fa6d89b011694955e469aa5bfb720307febd416f97f" exitCode=0 Oct 31 23:10:42.988065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.987901 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" event={"ID":"9578f643-4094-4d1d-be0d-5493f292a27d","Type":"ContainerDied","Data":"0dbc2776b0634116d6327fa6d89b011694955e469aa5bfb720307febd416f97f"} Oct 31 23:10:42.989663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.989636 5043 generic.go:356] "Generic (PLEG): container finished" podID="7e60aa67-7463-4fe2-8154-2ca468cbd804" containerID="49e62936a733cf2bd5b7b7c25fee9f0a86f7dbdf294432948e8b06f3c6a69566" exitCode=0 Oct 31 23:10:42.989786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.989721 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" event={"ID":"7e60aa67-7463-4fe2-8154-2ca468cbd804","Type":"ContainerDied","Data":"49e62936a733cf2bd5b7b7c25fee9f0a86f7dbdf294432948e8b06f3c6a69566"} Oct 31 23:10:42.991429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.991407 5043 generic.go:356] "Generic (PLEG): container finished" podID="fe928e72-e3a3-484c-8b4e-aa642aed31fc" containerID="9170ca7e774508a7ecde8b5f7f42147b60fbd120dda647844c4e7f9d03d8e87a" exitCode=0 Oct 31 23:10:42.991520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:42.991488 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" event={"ID":"fe928e72-e3a3-484c-8b4e-aa642aed31fc","Type":"ContainerDied","Data":"9170ca7e774508a7ecde8b5f7f42147b60fbd120dda647844c4e7f9d03d8e87a"} Oct 31 23:10:43.193254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.193216 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:43.332361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.332323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:43.338824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.338793 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5\" (UniqueName: \"kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5\") pod \"227acc28-b8eb-4ab9-a764-83b71dc868e8\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " Oct 31 23:10:43.339013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.338849 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca\") pod \"227acc28-b8eb-4ab9-a764-83b71dc868e8\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " Oct 31 23:10:43.339013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.338881 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca\") pod \"227acc28-b8eb-4ab9-a764-83b71dc868e8\" (UID: \"227acc28-b8eb-4ab9-a764-83b71dc868e8\") " Oct 31 23:10:43.340357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.339265 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca" pod "227acc28-b8eb-4ab9-a764-83b71dc868e8" (UID: "227acc28-b8eb-4ab9-a764-83b71dc868e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:43.344173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.343202 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5" pod "227acc28-b8eb-4ab9-a764-83b71dc868e8" (UID: "227acc28-b8eb-4ab9-a764-83b71dc868e8"). InnerVolumeSpecName "kube-api-access-h22z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:43.344319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.344012 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca" pod "227acc28-b8eb-4ab9-a764-83b71dc868e8" (UID: "227acc28-b8eb-4ab9-a764-83b71dc868e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:43.441106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.441067 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22z5\" (UniqueName: \"kubernetes.io/projected/227acc28-b8eb-4ab9-a764-83b71dc868e8-kube-api-access-h22z5\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:43.441106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.441107 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:43.441300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.441125 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227acc28-b8eb-4ab9-a764-83b71dc868e8-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:43.987464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.987425 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt"] Oct 31 23:10:43.987965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.987950 5043 cpu_manager.go:414] "containerMap: removing container" podUID="227acc28-b8eb-4ab9-a764-83b71dc868e8" containerName="revoke-keys" Oct 31 23:10:43.988028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.987966 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227acc28-b8eb-4ab9-a764-83b71dc868e8" containerName="revoke-keys" Oct 31 23:10:43.988081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.988065 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="227acc28-b8eb-4ab9-a764-83b71dc868e8" containerName="revoke-keys" Oct 31 23:10:43.991383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:43.991361 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.000166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.000102 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt"] Oct 31 23:10:44.002105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.002085 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" Oct 31 23:10:44.002208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.002117 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm" event={"ID":"227acc28-b8eb-4ab9-a764-83b71dc868e8","Type":"ContainerDied","Data":"b57107c09ee22573f4468dcb6f148480e9d74f520b24471f3307a30d3f4e3aa6"} Oct 31 23:10:44.002208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.002165 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b57107c09ee22573f4468dcb6f148480e9d74f520b24471f3307a30d3f4e3aa6" Oct 31 23:10:44.029296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.027003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:44.150939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.150909 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.151189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.151173 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.151339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.151329 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz4w\" (UniqueName: \"kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.252224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.252193 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz4w\" (UniqueName: \"kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.252360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.252289 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.253052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.253028 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.253213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.253196 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.255429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.255405 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.265382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.265357 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz4w\" (UniqueName: \"kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w\") pod \"maas-api-revoke-sub-1be0f84a-spzgt\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.286876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.286848 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:44.289199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.289172 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:44.292210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.292187 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:44.309887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.309860 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:44.455529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.455494 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn\" (UniqueName: \"kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn\") pod \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " Oct 31 23:10:44.455728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.455707 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca\") pod \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " Oct 31 23:10:44.455795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.455768 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca\") pod \"7e60aa67-7463-4fe2-8154-2ca468cbd804\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " Oct 31 23:10:44.455871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.455822 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca\") pod \"9578f643-4094-4d1d-be0d-5493f292a27d\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " Oct 31 23:10:44.455974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.455951 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca\") pod \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\" (UID: \"fe928e72-e3a3-484c-8b4e-aa642aed31fc\") " Oct 31 23:10:44.456109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456006 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca\") pod \"9578f643-4094-4d1d-be0d-5493f292a27d\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " Oct 31 23:10:44.456109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456050 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b\" (UniqueName: \"kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b\") pod \"7e60aa67-7463-4fe2-8154-2ca468cbd804\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " Oct 31 23:10:44.456109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456088 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca\") pod \"7e60aa67-7463-4fe2-8154-2ca468cbd804\" (UID: \"7e60aa67-7463-4fe2-8154-2ca468cbd804\") " Oct 31 23:10:44.456280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456120 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca" pod "fe928e72-e3a3-484c-8b4e-aa642aed31fc" (UID: "fe928e72-e3a3-484c-8b4e-aa642aed31fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:44.456280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456156 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5\" (UniqueName: \"kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5\") pod \"9578f643-4094-4d1d-be0d-5493f292a27d\" (UID: \"9578f643-4094-4d1d-be0d-5493f292a27d\") " Oct 31 23:10:44.456280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456203 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca" pod "7e60aa67-7463-4fe2-8154-2ca468cbd804" (UID: "7e60aa67-7463-4fe2-8154-2ca468cbd804"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:44.456280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456235 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca" pod "9578f643-4094-4d1d-be0d-5493f292a27d" (UID: "9578f643-4094-4d1d-be0d-5493f292a27d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:44.456720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456645 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.456720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456673 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.456720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.456688 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.458353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.458321 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn" pod "fe928e72-e3a3-484c-8b4e-aa642aed31fc" (UID: "fe928e72-e3a3-484c-8b4e-aa642aed31fc"). InnerVolumeSpecName "kube-api-access-bvckn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:44.458939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.458827 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca" pod "9578f643-4094-4d1d-be0d-5493f292a27d" (UID: "9578f643-4094-4d1d-be0d-5493f292a27d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:44.458939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.458919 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca" pod "fe928e72-e3a3-484c-8b4e-aa642aed31fc" (UID: "fe928e72-e3a3-484c-8b4e-aa642aed31fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:44.459139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.459119 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca" pod "7e60aa67-7463-4fe2-8154-2ca468cbd804" (UID: "7e60aa67-7463-4fe2-8154-2ca468cbd804"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:44.459202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.459167 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b" pod "7e60aa67-7463-4fe2-8154-2ca468cbd804" (UID: "7e60aa67-7463-4fe2-8154-2ca468cbd804"). InnerVolumeSpecName "kube-api-access-jcb8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:44.459328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.459304 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5" pod "9578f643-4094-4d1d-be0d-5493f292a27d" (UID: "9578f643-4094-4d1d-be0d-5493f292a27d"). InnerVolumeSpecName "kube-api-access-qlwv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:44.558369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.558328 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcb8b\" (UniqueName: \"kubernetes.io/projected/7e60aa67-7463-4fe2-8154-2ca468cbd804-kube-api-access-jcb8b\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.558369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.558366 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60aa67-7463-4fe2-8154-2ca468cbd804-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.558602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.558382 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlwv5\" (UniqueName: \"kubernetes.io/projected/9578f643-4094-4d1d-be0d-5493f292a27d-kube-api-access-qlwv5\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.558602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.558396 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvckn\" (UniqueName: \"kubernetes.io/projected/fe928e72-e3a3-484c-8b4e-aa642aed31fc-kube-api-access-bvckn\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.558602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.558412 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe928e72-e3a3-484c-8b4e-aa642aed31fc-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.558602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.558426 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9578f643-4094-4d1d-be0d-5493f292a27d-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:44.611391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.611363 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt"] Oct 31 23:10:44.612202 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:44.612172 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eeddfa8_e552_4ed6_a6d7_faeca208530a.slice/crio-55b085a48d04cc7ab42e9a5c5fe7028414e81e968d1d015d9f0488a9af48b467 WatchSource:0}: Error finding container 55b085a48d04cc7ab42e9a5c5fe7028414e81e968d1d015d9f0488a9af48b467: Status 404 returned error can't find the container with id 55b085a48d04cc7ab42e9a5c5fe7028414e81e968d1d015d9f0488a9af48b467 Oct 31 23:10:44.821529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:44.821434 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:45.009794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.009766 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" Oct 31 23:10:45.010227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.009762 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt" event={"ID":"9578f643-4094-4d1d-be0d-5493f292a27d","Type":"ContainerDied","Data":"17569d9971bc72f4888ad3a6c0a095cd23d369318051a9eab2d4995675c9caf1"} Oct 31 23:10:45.010227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.009914 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17569d9971bc72f4888ad3a6c0a095cd23d369318051a9eab2d4995675c9caf1" Oct 31 23:10:45.011230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.011197 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" event={"ID":"7e60aa67-7463-4fe2-8154-2ca468cbd804","Type":"ContainerDied","Data":"40c65ee7eb8ab2b0ba4b19d5f30004f30de6bbe0eba4f9275fb38c2c142cdbbf"} Oct 31 23:10:45.011375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.011247 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40c65ee7eb8ab2b0ba4b19d5f30004f30de6bbe0eba4f9275fb38c2c142cdbbf" Oct 31 23:10:45.011375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.011211 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r" Oct 31 23:10:45.013036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.013011 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" event={"ID":"5eeddfa8-e552-4ed6-a6d7-faeca208530a","Type":"ContainerStarted","Data":"c700e343e13353fb9d37df3bc3d573ed694a5ac4180842f99bb2c68c463256dc"} Oct 31 23:10:45.013161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.013053 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" event={"ID":"5eeddfa8-e552-4ed6-a6d7-faeca208530a","Type":"ContainerStarted","Data":"55b085a48d04cc7ab42e9a5c5fe7028414e81e968d1d015d9f0488a9af48b467"} Oct 31 23:10:45.014641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.014614 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" event={"ID":"fe928e72-e3a3-484c-8b4e-aa642aed31fc","Type":"ContainerDied","Data":"2010e8be7eb011bff895c29e65c3edb56f246582e20b4c63c7c9a5999981d2b0"} Oct 31 23:10:45.014641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.014635 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87" Oct 31 23:10:45.014824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.014658 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2010e8be7eb011bff895c29e65c3edb56f246582e20b4c63c7c9a5999981d2b0" Oct 31 23:10:45.035674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.035629 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" podStartSLOduration=2.035616487 podStartE2EDuration="2.035616487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:45.031942881 +0000 UTC m=+1271.261497292" watchObservedRunningTime="2026-10-31 23:10:45.035616487 +0000 UTC m=+1271.265170932" Oct 31 23:10:45.516791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:45.516744 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:46.025375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:46.025340 5043 generic.go:356] "Generic (PLEG): container finished" podID="5eeddfa8-e552-4ed6-a6d7-faeca208530a" containerID="c700e343e13353fb9d37df3bc3d573ed694a5ac4180842f99bb2c68c463256dc" exitCode=0 Oct 31 23:10:46.025842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:46.025387 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" event={"ID":"5eeddfa8-e552-4ed6-a6d7-faeca208530a","Type":"ContainerDied","Data":"c700e343e13353fb9d37df3bc3d573ed694a5ac4180842f99bb2c68c463256dc"} Oct 31 23:10:46.128011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:46.127200 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:46.796498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:46.796464 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:47.205402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.205373 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:47.392542 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.392464 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w\" (UniqueName: \"kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w\") pod \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " Oct 31 23:10:47.392542 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.392512 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca\") pod \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " Oct 31 23:10:47.392747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.392602 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca\") pod \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\" (UID: \"5eeddfa8-e552-4ed6-a6d7-faeca208530a\") " Oct 31 23:10:47.392930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.392909 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca" pod "5eeddfa8-e552-4ed6-a6d7-faeca208530a" (UID: "5eeddfa8-e552-4ed6-a6d7-faeca208530a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:47.393103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.393086 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:47.394706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.394679 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w" pod "5eeddfa8-e552-4ed6-a6d7-faeca208530a" (UID: "5eeddfa8-e552-4ed6-a6d7-faeca208530a"). InnerVolumeSpecName "kube-api-access-pgz4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:47.394790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.394754 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca" pod "5eeddfa8-e552-4ed6-a6d7-faeca208530a" (UID: "5eeddfa8-e552-4ed6-a6d7-faeca208530a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:47.416384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.416355 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:47.495360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.495328 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgz4w\" (UniqueName: \"kubernetes.io/projected/5eeddfa8-e552-4ed6-a6d7-faeca208530a-kube-api-access-pgz4w\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:47.495360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:47.495357 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eeddfa8-e552-4ed6-a6d7-faeca208530a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:48.040099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:48.040061 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" Oct 31 23:10:48.040099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:48.040073 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt" event={"ID":"5eeddfa8-e552-4ed6-a6d7-faeca208530a","Type":"ContainerDied","Data":"55b085a48d04cc7ab42e9a5c5fe7028414e81e968d1d015d9f0488a9af48b467"} Oct 31 23:10:48.040357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:48.040128 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55b085a48d04cc7ab42e9a5c5fe7028414e81e968d1d015d9f0488a9af48b467" Oct 31 23:10:48.110196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:48.110157 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:48.722431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:48.722389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:49.406918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:49.406877 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:50.144523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:50.144474 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:50.799528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:50.799491 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:51.480994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:51.480957 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:52.224833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.224796 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w"] Oct 31 23:10:52.225374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225358 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7e60aa67-7463-4fe2-8154-2ca468cbd804" containerName="revoke-keys" Oct 31 23:10:52.225423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225376 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e60aa67-7463-4fe2-8154-2ca468cbd804" containerName="revoke-keys" Oct 31 23:10:52.225423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225390 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9578f643-4094-4d1d-be0d-5493f292a27d" containerName="revoke-keys" Oct 31 23:10:52.225423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225395 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9578f643-4094-4d1d-be0d-5493f292a27d" containerName="revoke-keys" Oct 31 23:10:52.225423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225406 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5eeddfa8-e552-4ed6-a6d7-faeca208530a" containerName="revoke-keys" Oct 31 23:10:52.225423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225410 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eeddfa8-e552-4ed6-a6d7-faeca208530a" containerName="revoke-keys" Oct 31 23:10:52.225655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225426 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fe928e72-e3a3-484c-8b4e-aa642aed31fc" containerName="revoke-keys" Oct 31 23:10:52.225655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225430 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe928e72-e3a3-484c-8b4e-aa642aed31fc" containerName="revoke-keys" Oct 31 23:10:52.225655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225502 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e60aa67-7463-4fe2-8154-2ca468cbd804" containerName="revoke-keys" Oct 31 23:10:52.225655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225509 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9578f643-4094-4d1d-be0d-5493f292a27d" containerName="revoke-keys" Oct 31 23:10:52.225655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225515 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eeddfa8-e552-4ed6-a6d7-faeca208530a" containerName="revoke-keys" Oct 31 23:10:52.225655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.225522 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe928e72-e3a3-484c-8b4e-aa642aed31fc" containerName="revoke-keys" Oct 31 23:10:52.228863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.228844 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.232985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.232962 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:10:52.233898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.233872 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:10:52.234010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.233970 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:10:52.234070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.234044 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:10:52.236476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.236444 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w"] Oct 31 23:10:52.304391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.304354 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.304391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.304395 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzzf\" (UniqueName: \"kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.304698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.304674 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.307837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.307812 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv"] Oct 31 23:10:52.312530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.312505 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.319070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.319043 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv"] Oct 31 23:10:52.333743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.330036 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:52.405734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.405692 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.405924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.405852 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mnv8\" (UniqueName: \"kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.405924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.405907 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.406010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.405978 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.406101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.406087 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.406150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.406109 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzzf\" (UniqueName: \"kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.406812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.406790 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.408240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.408215 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.419727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.419697 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzzf\" (UniqueName: \"kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf\") pod \"maas-api-revoke-sub-74a9cf17-nbj6w\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.507758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.507722 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.508201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.507847 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mnv8\" (UniqueName: \"kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.508201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.507938 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.508668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.508644 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.510214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.510190 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.520224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.520199 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mnv8\" (UniqueName: \"kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8\") pod \"maas-api-revoke-sub-4fbca134-5tmrv\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.544748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.544713 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:52.655886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.655856 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:52.879342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.879308 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w"] Oct 31 23:10:52.880393 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:52.880363 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46bdbaf_4f06_4e52_900b_74f15e5f4c4e.slice/crio-00b2eb147d4f05273abae95c30f53ee1da66fce29694f850416889d5e133dd6a WatchSource:0}: Error finding container 00b2eb147d4f05273abae95c30f53ee1da66fce29694f850416889d5e133dd6a: Status 404 returned error can't find the container with id 00b2eb147d4f05273abae95c30f53ee1da66fce29694f850416889d5e133dd6a Oct 31 23:10:52.952055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:52.952031 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv"] Oct 31 23:10:52.953541 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:52.953514 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1982c4_16ae_453d_b617_8e69cead1b3a.slice/crio-2d261f2da340b67bf08f1bbf450f8ed23ec6ef91809ca0ecbb9d4cc306fb0188 WatchSource:0}: Error finding container 2d261f2da340b67bf08f1bbf450f8ed23ec6ef91809ca0ecbb9d4cc306fb0188: Status 404 returned error can't find the container with id 2d261f2da340b67bf08f1bbf450f8ed23ec6ef91809ca0ecbb9d4cc306fb0188 Oct 31 23:10:53.125360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:53.125318 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" event={"ID":"cb1982c4-16ae-453d-b617-8e69cead1b3a","Type":"ContainerStarted","Data":"2d261f2da340b67bf08f1bbf450f8ed23ec6ef91809ca0ecbb9d4cc306fb0188"} Oct 31 23:10:53.126991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:53.126932 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" event={"ID":"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e","Type":"ContainerStarted","Data":"00b2eb147d4f05273abae95c30f53ee1da66fce29694f850416889d5e133dd6a"} Oct 31 23:10:53.306216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:53.306099 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:54.004872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.004835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:54.140341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.140301 5043 generic.go:356] "Generic (PLEG): container finished" podID="cb1982c4-16ae-453d-b617-8e69cead1b3a" containerID="64ef7970708d5524eadbaca989e3b2936a69d1b1ca2acea876a74af987040206" exitCode=0 Oct 31 23:10:54.140587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.140441 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" event={"ID":"cb1982c4-16ae-453d-b617-8e69cead1b3a","Type":"ContainerDied","Data":"64ef7970708d5524eadbaca989e3b2936a69d1b1ca2acea876a74af987040206"} Oct 31 23:10:54.142964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.142932 5043 generic.go:356] "Generic (PLEG): container finished" podID="b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" containerID="6c841bf719b9c5b6096d6bd36f2edf4577f405bb9989a5cfca19089054a5be7b" exitCode=0 Oct 31 23:10:54.143140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.143122 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" event={"ID":"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e","Type":"ContainerDied","Data":"6c841bf719b9c5b6096d6bd36f2edf4577f405bb9989a5cfca19089054a5be7b"} Oct 31 23:10:54.397081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.397000 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5"] Oct 31 23:10:54.400834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.400816 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.409987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.409963 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5"] Oct 31 23:10:54.433467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.433431 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm"] Oct 31 23:10:54.439405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.439347 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.439684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.439658 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm"] Oct 31 23:10:54.533786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.533752 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkbl\" (UniqueName: \"kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.533971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.533892 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.534024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.533998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slh2t\" (UniqueName: \"kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.534024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.534016 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.534125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.534069 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.534195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.534174 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.635616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.635575 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.635829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.635655 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.635829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.635747 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkbl\" (UniqueName: \"kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.635960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.635848 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.635960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.635953 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slh2t\" (UniqueName: \"kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.636080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.635976 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.636576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.636536 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.636686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.636537 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.638848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.638821 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.638967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.638953 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.650781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.650687 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkbl\" (UniqueName: \"kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl\") pod \"maas-api-revoke-sub-84657d16-7cxpm\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.650901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.650823 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slh2t\" (UniqueName: \"kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t\") pod \"maas-api-revoke-sub-7c18ddf6-xdvp5\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.654756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.654730 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7"] Oct 31 23:10:54.660058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.660036 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.676695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.676592 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7"] Oct 31 23:10:54.717056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.717030 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:54.737496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.737465 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.737685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.737660 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.737758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.737703 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.737817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.737792 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.737877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.737862 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.737970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.737956 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmzb\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.738064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.738048 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.738285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.738110 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.738285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.738237 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.759299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.759262 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:54.812491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.812453 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.849744 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.849801 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.849903 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.849933 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.850030 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.850059 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.850090 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.850196 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.850238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmzb\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.854055 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.854325 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.856369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.854861 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.858296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.856443 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.858296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.857316 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.858296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.857477 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.858296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.857624 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.865767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.865739 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmzb\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.871632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.867693 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:54.986902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:54.986813 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:55.004112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.004087 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5"] Oct 31 23:10:55.005866 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:55.005841 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0499052f_9164_4ef0_b9d1_fc4ee4168900.slice/crio-58959197ef1d71bf26097361e71d78edfc3808a85fa7fc621cdf5d30cffea8e4 WatchSource:0}: Error finding container 58959197ef1d71bf26097361e71d78edfc3808a85fa7fc621cdf5d30cffea8e4: Status 404 returned error can't find the container with id 58959197ef1d71bf26097361e71d78edfc3808a85fa7fc621cdf5d30cffea8e4 Oct 31 23:10:55.097104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.096346 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm"] Oct 31 23:10:55.151192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.151160 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" event={"ID":"1a97f61d-f903-42dc-ba88-ba0d98cd77b5","Type":"ContainerStarted","Data":"c40625e7fde62b62ddbd5481b6e750a24c78a19e884606644dcc0031522441d9"} Oct 31 23:10:55.152455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.152433 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" event={"ID":"0499052f-9164-4ef0-b9d1-fc4ee4168900","Type":"ContainerStarted","Data":"58959197ef1d71bf26097361e71d78edfc3808a85fa7fc621cdf5d30cffea8e4"} Oct 31 23:10:55.285325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.285287 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7"] Oct 31 23:10:55.291165 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:55.291130 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fdaed9_d6a5_4748_b93a_d385a615164a.slice/crio-523642639beff6201d3902921117776313b6fce098f7a2e3a8d8a8f02fc63fc0 WatchSource:0}: Error finding container 523642639beff6201d3902921117776313b6fce098f7a2e3a8d8a8f02fc63fc0: Status 404 returned error can't find the container with id 523642639beff6201d3902921117776313b6fce098f7a2e3a8d8a8f02fc63fc0 Oct 31 23:10:55.296572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.293958 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:10:55.296572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.294046 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:10:55.296572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.294087 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:10:55.425580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.425537 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:55.430194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.430170 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:55.559298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.559240 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca\") pod \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " Oct 31 23:10:55.559489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.559372 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf\" (UniqueName: \"kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf\") pod \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " Oct 31 23:10:55.559489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.559422 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8\" (UniqueName: \"kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8\") pod \"cb1982c4-16ae-453d-b617-8e69cead1b3a\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " Oct 31 23:10:55.559653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.559486 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca\") pod \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\" (UID: \"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e\") " Oct 31 23:10:55.559704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.559659 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca\") pod \"cb1982c4-16ae-453d-b617-8e69cead1b3a\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " Oct 31 23:10:55.560114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.559758 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca\") pod \"cb1982c4-16ae-453d-b617-8e69cead1b3a\" (UID: \"cb1982c4-16ae-453d-b617-8e69cead1b3a\") " Oct 31 23:10:55.560469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.560439 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca" pod "b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" (UID: "b46bdbaf-4f06-4e52-900b-74f15e5f4c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:55.561675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.561645 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca" pod "cb1982c4-16ae-453d-b617-8e69cead1b3a" (UID: "cb1982c4-16ae-453d-b617-8e69cead1b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:55.565295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.565253 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca" pod "b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" (UID: "b46bdbaf-4f06-4e52-900b-74f15e5f4c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:55.565514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.565498 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8" pod "cb1982c4-16ae-453d-b617-8e69cead1b3a" (UID: "cb1982c4-16ae-453d-b617-8e69cead1b3a"). InnerVolumeSpecName "kube-api-access-5mnv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:55.567368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.567337 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca" pod "cb1982c4-16ae-453d-b617-8e69cead1b3a" (UID: "cb1982c4-16ae-453d-b617-8e69cead1b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:55.567472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.567391 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf" pod "b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" (UID: "b46bdbaf-4f06-4e52-900b-74f15e5f4c4e"). InnerVolumeSpecName "kube-api-access-gpzzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:55.628572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.627109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:55.661501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.661463 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:55.661702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.661505 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpzzf\" (UniqueName: \"kubernetes.io/projected/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-kube-api-access-gpzzf\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:55.661702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.661523 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mnv8\" (UniqueName: \"kubernetes.io/projected/cb1982c4-16ae-453d-b617-8e69cead1b3a-kube-api-access-5mnv8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:55.661702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.661537 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:55.661702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.661576 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:55.661702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.661590 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1982c4-16ae-453d-b617-8e69cead1b3a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:55.853311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853230 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q"] Oct 31 23:10:55.853802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853785 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" containerName="revoke-keys" Oct 31 23:10:55.853854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853803 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" containerName="revoke-keys" Oct 31 23:10:55.853854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853817 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cb1982c4-16ae-453d-b617-8e69cead1b3a" containerName="revoke-keys" Oct 31 23:10:55.853854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853822 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1982c4-16ae-453d-b617-8e69cead1b3a" containerName="revoke-keys" Oct 31 23:10:55.853968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853901 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1982c4-16ae-453d-b617-8e69cead1b3a" containerName="revoke-keys" Oct 31 23:10:55.853968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.853917 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" containerName="revoke-keys" Oct 31 23:10:55.857566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.857518 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:55.866169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.866141 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q"] Oct 31 23:10:55.964965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.964930 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:55.964965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.964969 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szz5j\" (UniqueName: \"kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:55.965199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:55.965117 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.067037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.067008 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.067474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.067050 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szz5j\" (UniqueName: \"kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.067474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.067173 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.067910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.067881 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.069351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.069332 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.079490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.079468 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szz5j\" (UniqueName: \"kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j\") pod \"maas-api-revoke-sub-1f0abaa7-9nl5q\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.162671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.162630 5043 generic.go:356] "Generic (PLEG): container finished" podID="0499052f-9164-4ef0-b9d1-fc4ee4168900" containerID="b410ae72c0e082bd54b5919cfcac129054251986343114860294b6e2274565ff" exitCode=0 Oct 31 23:10:56.162851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.162715 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" event={"ID":"0499052f-9164-4ef0-b9d1-fc4ee4168900","Type":"ContainerDied","Data":"b410ae72c0e082bd54b5919cfcac129054251986343114860294b6e2274565ff"} Oct 31 23:10:56.164437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.164414 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" Oct 31 23:10:56.164592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.164460 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv" event={"ID":"cb1982c4-16ae-453d-b617-8e69cead1b3a","Type":"ContainerDied","Data":"2d261f2da340b67bf08f1bbf450f8ed23ec6ef91809ca0ecbb9d4cc306fb0188"} Oct 31 23:10:56.164592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.164517 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d261f2da340b67bf08f1bbf450f8ed23ec6ef91809ca0ecbb9d4cc306fb0188" Oct 31 23:10:56.166868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.166208 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" Oct 31 23:10:56.166868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.166235 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w" event={"ID":"b46bdbaf-4f06-4e52-900b-74f15e5f4c4e","Type":"ContainerDied","Data":"00b2eb147d4f05273abae95c30f53ee1da66fce29694f850416889d5e133dd6a"} Oct 31 23:10:56.166868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.166275 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00b2eb147d4f05273abae95c30f53ee1da66fce29694f850416889d5e133dd6a" Oct 31 23:10:56.171762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.170022 5043 generic.go:356] "Generic (PLEG): container finished" podID="1a97f61d-f903-42dc-ba88-ba0d98cd77b5" containerID="7611d804104cff17f9976f7c57186e5997df0521d0b89e04b15ace0326cbf0c0" exitCode=0 Oct 31 23:10:56.171762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.170094 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" event={"ID":"1a97f61d-f903-42dc-ba88-ba0d98cd77b5","Type":"ContainerDied","Data":"7611d804104cff17f9976f7c57186e5997df0521d0b89e04b15ace0326cbf0c0"} Oct 31 23:10:56.172813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.172782 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" event={"ID":"f5fdaed9-d6a5-4748-b93a-d385a615164a","Type":"ContainerStarted","Data":"62774c08ba9a484dd0ebc15316de6103802c19591ca8999ef4eeadc376445b46"} Oct 31 23:10:56.172897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.172831 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" event={"ID":"f5fdaed9-d6a5-4748-b93a-d385a615164a","Type":"ContainerStarted","Data":"523642639beff6201d3902921117776313b6fce098f7a2e3a8d8a8f02fc63fc0"} Oct 31 23:10:56.176265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.176023 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:56.236191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.236119 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podStartSLOduration=2.23609809 podStartE2EDuration="2.23609809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:56.231888827 +0000 UTC m=+1282.461443250" watchObservedRunningTime="2026-10-31 23:10:56.23609809 +0000 UTC m=+1282.465652506" Oct 31 23:10:56.518779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.513458 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q"] Oct 31 23:10:56.534596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.529126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:56.987784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.987757 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:10:56.989235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.989215 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:10:56.989349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:56.989261 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:10:57.109039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.109003 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2"] Oct 31 23:10:57.117577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.113660 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.165685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.165654 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2"] Oct 31 23:10:57.181575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.181456 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.181701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.181611 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.181701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.181630 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44x9\" (UniqueName: \"kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.193598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.184055 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" event={"ID":"62a0d684-84d4-4e5a-ad73-5e3615604a98","Type":"ContainerStarted","Data":"62a66558c63d7fac5b8012c0b89c3040426f91d0da7d776839378ae1d2e270df"} Oct 31 23:10:57.193598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.184116 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" event={"ID":"62a0d684-84d4-4e5a-ad73-5e3615604a98","Type":"ContainerStarted","Data":"9497f75da33d8e7f503545dc16035a61ba7149e849e053d7912e5addee9b6123"} Oct 31 23:10:57.227624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.222835 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" podStartSLOduration=2.222812888 podStartE2EDuration="2.222812888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:10:57.20994414 +0000 UTC m=+1283.439498557" watchObservedRunningTime="2026-10-31 23:10:57.222812888 +0000 UTC m=+1283.452367301" Oct 31 23:10:57.283266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.283234 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.283391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.283271 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44x9\" (UniqueName: \"kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.283391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.283317 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.284824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.284228 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.289573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.288157 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.291759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.291601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:57.302785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.302719 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44x9\" (UniqueName: \"kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9\") pod \"maas-api-revoke-sub-0f47faab-ll7p2\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.472776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.472347 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:10:57.492595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.492566 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:57.581862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.581825 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:57.586323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.586290 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca\") pod \"0499052f-9164-4ef0-b9d1-fc4ee4168900\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " Oct 31 23:10:57.586441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.586378 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t\" (UniqueName: \"kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t\") pod \"0499052f-9164-4ef0-b9d1-fc4ee4168900\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " Oct 31 23:10:57.586441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.586413 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca\") pod \"0499052f-9164-4ef0-b9d1-fc4ee4168900\" (UID: \"0499052f-9164-4ef0-b9d1-fc4ee4168900\") " Oct 31 23:10:57.588412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.588382 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca" pod "0499052f-9164-4ef0-b9d1-fc4ee4168900" (UID: "0499052f-9164-4ef0-b9d1-fc4ee4168900"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:57.591655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.591631 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t" pod "0499052f-9164-4ef0-b9d1-fc4ee4168900" (UID: "0499052f-9164-4ef0-b9d1-fc4ee4168900"). InnerVolumeSpecName "kube-api-access-slh2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:57.591763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.591727 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca" pod "0499052f-9164-4ef0-b9d1-fc4ee4168900" (UID: "0499052f-9164-4ef0-b9d1-fc4ee4168900"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:57.689381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.689349 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl\" (UniqueName: \"kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl\") pod \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " Oct 31 23:10:57.689577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.689392 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca\") pod \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " Oct 31 23:10:57.689577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.689445 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca\") pod \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\" (UID: \"1a97f61d-f903-42dc-ba88-ba0d98cd77b5\") " Oct 31 23:10:57.689706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.689680 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:57.689706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.689692 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slh2t\" (UniqueName: \"kubernetes.io/projected/0499052f-9164-4ef0-b9d1-fc4ee4168900-kube-api-access-slh2t\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:57.689706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.689702 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0499052f-9164-4ef0-b9d1-fc4ee4168900-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:57.690322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.690293 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca" pod "1a97f61d-f903-42dc-ba88-ba0d98cd77b5" (UID: "1a97f61d-f903-42dc-ba88-ba0d98cd77b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:57.692472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.692445 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca" pod "1a97f61d-f903-42dc-ba88-ba0d98cd77b5" (UID: "1a97f61d-f903-42dc-ba88-ba0d98cd77b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:57.693270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.693241 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl" pod "1a97f61d-f903-42dc-ba88-ba0d98cd77b5" (UID: "1a97f61d-f903-42dc-ba88-ba0d98cd77b5"). InnerVolumeSpecName "kube-api-access-8jkbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:57.790987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.790955 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkbl\" (UniqueName: \"kubernetes.io/projected/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-kube-api-access-8jkbl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:57.790987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.790985 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:57.791174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.790997 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a97f61d-f903-42dc-ba88-ba0d98cd77b5-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:57.803196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.803164 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2"] Oct 31 23:10:57.812597 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:10:57.812572 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3543cacb_852e_4374_840f_03acd54a50e8.slice/crio-06be3ab2c269e895fc84594e20f3a92b85dfa805fb0cc2ab6c813b8cd935ffad WatchSource:0}: Error finding container 06be3ab2c269e895fc84594e20f3a92b85dfa805fb0cc2ab6c813b8cd935ffad: Status 404 returned error can't find the container with id 06be3ab2c269e895fc84594e20f3a92b85dfa805fb0cc2ab6c813b8cd935ffad Oct 31 23:10:57.987422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.987396 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:10:57.987573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:57.987445 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:10:58.035725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.032275 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:58.192122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.192086 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" event={"ID":"3543cacb-852e-4374-840f-03acd54a50e8","Type":"ContainerStarted","Data":"06be3ab2c269e895fc84594e20f3a92b85dfa805fb0cc2ab6c813b8cd935ffad"} Oct 31 23:10:58.193528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.193506 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" Oct 31 23:10:58.193702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.193572 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5" event={"ID":"0499052f-9164-4ef0-b9d1-fc4ee4168900","Type":"ContainerDied","Data":"58959197ef1d71bf26097361e71d78edfc3808a85fa7fc621cdf5d30cffea8e4"} Oct 31 23:10:58.193702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.193626 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58959197ef1d71bf26097361e71d78edfc3808a85fa7fc621cdf5d30cffea8e4" Oct 31 23:10:58.195339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.195314 5043 generic.go:356] "Generic (PLEG): container finished" podID="62a0d684-84d4-4e5a-ad73-5e3615604a98" containerID="62a66558c63d7fac5b8012c0b89c3040426f91d0da7d776839378ae1d2e270df" exitCode=0 Oct 31 23:10:58.195453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.195348 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" event={"ID":"62a0d684-84d4-4e5a-ad73-5e3615604a98","Type":"ContainerDied","Data":"62a66558c63d7fac5b8012c0b89c3040426f91d0da7d776839378ae1d2e270df"} Oct 31 23:10:58.196860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.196844 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" Oct 31 23:10:58.196971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.196894 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm" event={"ID":"1a97f61d-f903-42dc-ba88-ba0d98cd77b5","Type":"ContainerDied","Data":"c40625e7fde62b62ddbd5481b6e750a24c78a19e884606644dcc0031522441d9"} Oct 31 23:10:58.196971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.196928 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c40625e7fde62b62ddbd5481b6e750a24c78a19e884606644dcc0031522441d9" Oct 31 23:10:58.753285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.751208 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:58.987482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.987451 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:10:58.987619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:58.987505 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:10:59.204419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.204332 5043 generic.go:356] "Generic (PLEG): container finished" podID="3543cacb-852e-4374-840f-03acd54a50e8" containerID="5106288b7536c4f1f280665ba2bb4499854d26e0e7155157531d64bdb16b75a5" exitCode=0 Oct 31 23:10:59.204419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.204360 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" event={"ID":"3543cacb-852e-4374-840f-03acd54a50e8","Type":"ContainerDied","Data":"5106288b7536c4f1f280665ba2bb4499854d26e0e7155157531d64bdb16b75a5"} Oct 31 23:10:59.445067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.445031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:10:59.448828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.448805 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:10:59.512297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.512268 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca\") pod \"62a0d684-84d4-4e5a-ad73-5e3615604a98\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " Oct 31 23:10:59.512618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.512601 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j\" (UniqueName: \"kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j\") pod \"62a0d684-84d4-4e5a-ad73-5e3615604a98\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " Oct 31 23:10:59.512780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.512766 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca\") pod \"62a0d684-84d4-4e5a-ad73-5e3615604a98\" (UID: \"62a0d684-84d4-4e5a-ad73-5e3615604a98\") " Oct 31 23:10:59.512925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.512675 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca" pod "62a0d684-84d4-4e5a-ad73-5e3615604a98" (UID: "62a0d684-84d4-4e5a-ad73-5e3615604a98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:10:59.513394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.513374 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:59.536568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.528073 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j" pod "62a0d684-84d4-4e5a-ad73-5e3615604a98" (UID: "62a0d684-84d4-4e5a-ad73-5e3615604a98"). InnerVolumeSpecName "kube-api-access-szz5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:10:59.565298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.564795 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca" pod "62a0d684-84d4-4e5a-ad73-5e3615604a98" (UID: "62a0d684-84d4-4e5a-ad73-5e3615604a98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:10:59.664568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.663501 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szz5j\" (UniqueName: \"kubernetes.io/projected/62a0d684-84d4-4e5a-ad73-5e3615604a98-kube-api-access-szz5j\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:59.664568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.663564 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a0d684-84d4-4e5a-ad73-5e3615604a98-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:10:59.988071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.988030 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:10:59.988256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:10:59.988103 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:11:00.145856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:00.145818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:00.253621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:00.241236 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" Oct 31 23:11:00.253621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:00.241849 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q" event={"ID":"62a0d684-84d4-4e5a-ad73-5e3615604a98","Type":"ContainerDied","Data":"9497f75da33d8e7f503545dc16035a61ba7149e849e053d7912e5addee9b6123"} Oct 31 23:11:00.253621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:00.241895 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9497f75da33d8e7f503545dc16035a61ba7149e849e053d7912e5addee9b6123" Oct 31 23:11:00.988061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:00.988021 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:00.988360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:00.988332 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:11:01.557652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.536224 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:11:01.557652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.554619 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:01.565436 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.565392 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:01.611736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.604813 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:01.612081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.612041 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:01.642093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.642054 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:01.832252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.832175 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:11:01.847966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.847838 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca\") pod \"3543cacb-852e-4374-840f-03acd54a50e8\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " Oct 31 23:11:01.848275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.847971 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9\" (UniqueName: \"kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9\") pod \"3543cacb-852e-4374-840f-03acd54a50e8\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " Oct 31 23:11:01.848275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.848027 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca\") pod \"3543cacb-852e-4374-840f-03acd54a50e8\" (UID: \"3543cacb-852e-4374-840f-03acd54a50e8\") " Oct 31 23:11:01.848854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.848824 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca" pod "3543cacb-852e-4374-840f-03acd54a50e8" (UID: "3543cacb-852e-4374-840f-03acd54a50e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:01.858772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.858708 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca" pod "3543cacb-852e-4374-840f-03acd54a50e8" (UID: "3543cacb-852e-4374-840f-03acd54a50e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:01.884792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.884750 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9" pod "3543cacb-852e-4374-840f-03acd54a50e8" (UID: "3543cacb-852e-4374-840f-03acd54a50e8"). InnerVolumeSpecName "kube-api-access-g44x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:01.966152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.966108 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:01.967055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.967030 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g44x9\" (UniqueName: \"kubernetes.io/projected/3543cacb-852e-4374-840f-03acd54a50e8-kube-api-access-g44x9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:01.967216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.967202 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3543cacb-852e-4374-840f-03acd54a50e8-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:01.998409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.994447 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:01.998409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:01.994511 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:11:02.447571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.415196 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:02.556748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.556684 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:11:02.557600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.557575 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:11:02.591655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.571504 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:02.603604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.597834 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:11:02.627017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.626971 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:02.638570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.631466 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:11:02.780581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.645171 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:11:02.928633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.761721 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:02.974570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.803749 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:11:02.974570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.974518 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:11:02.974829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.974620 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" event={"ID":"3543cacb-852e-4374-840f-03acd54a50e8","Type":"ContainerDied","Data":"06be3ab2c269e895fc84594e20f3a92b85dfa805fb0cc2ab6c813b8cd935ffad"} Oct 31 23:11:02.974829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.974658 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06be3ab2c269e895fc84594e20f3a92b85dfa805fb0cc2ab6c813b8cd935ffad" Oct 31 23:11:03.016681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:02.912844 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2" Oct 31 23:11:03.016681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.014140 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:03.141567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.140709 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:03.141567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.140773 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:11:03.561738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.561692 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:11:03.600611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.599629 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:03.634690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.634637 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:03.822250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.822107 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:03.915361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.914625 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:03.975679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.974642 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:03.987253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.987213 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 23:11:03.987495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:03.987466 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 23:11:04.296708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.296661 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:04.583121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.581630 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:11:04.596935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.596629 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:04.602341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.601635 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:04.636753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.635619 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:04.697224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.688914 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:11:04.707764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.707732 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck"] Oct 31 23:11:04.708811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.708784 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0499052f-9164-4ef0-b9d1-fc4ee4168900" containerName="revoke-keys" Oct 31 23:11:04.708972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.708957 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0499052f-9164-4ef0-b9d1-fc4ee4168900" containerName="revoke-keys" Oct 31 23:11:04.709088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709076 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1a97f61d-f903-42dc-ba88-ba0d98cd77b5" containerName="revoke-keys" Oct 31 23:11:04.709169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709158 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a97f61d-f903-42dc-ba88-ba0d98cd77b5" containerName="revoke-keys" Oct 31 23:11:04.709312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709299 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3543cacb-852e-4374-840f-03acd54a50e8" containerName="revoke-keys" Oct 31 23:11:04.709400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709388 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3543cacb-852e-4374-840f-03acd54a50e8" containerName="revoke-keys" Oct 31 23:11:04.709508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709496 5043 cpu_manager.go:414] "containerMap: removing container" podUID="62a0d684-84d4-4e5a-ad73-5e3615604a98" containerName="revoke-keys" Oct 31 23:11:04.709615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709603 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a0d684-84d4-4e5a-ad73-5e3615604a98" containerName="revoke-keys" Oct 31 23:11:04.709826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709810 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a97f61d-f903-42dc-ba88-ba0d98cd77b5" containerName="revoke-keys" Oct 31 23:11:04.709937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.709923 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3543cacb-852e-4374-840f-03acd54a50e8" containerName="revoke-keys" Oct 31 23:11:04.710044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.710032 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a0d684-84d4-4e5a-ad73-5e3615604a98" containerName="revoke-keys" Oct 31 23:11:04.710135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.710124 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0499052f-9164-4ef0-b9d1-fc4ee4168900" containerName="revoke-keys" Oct 31 23:11:04.740241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.740208 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:04.774681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.772941 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:11:04.775850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.775811 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:11:04.803599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.789926 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:11:04.803599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.790041 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:11:04.835865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.835753 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:11:04.889686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.875475 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw77m\" (UniqueName: \"kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:04.889686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.875537 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:04.889686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.875591 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:04.929643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.929599 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck"] Oct 31 23:11:04.937472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.937394 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:04.977605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.977356 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw77m\" (UniqueName: \"kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:04.977605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.977422 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:04.977605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.977458 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:05.004580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:04.978407 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:05.114742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.077220 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:05.217861 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.217824 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw77m\" (UniqueName: \"kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m\") pod \"maas-api-revoke-sub-fd82abeb-2ccck\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:05.242561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.239144 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:05.279437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.279345 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:11:05.307433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.298043 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh"] Oct 31 23:11:05.307433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.304396 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:05.376041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.375917 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:11:05.376301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.376229 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:11:05.376301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.376271 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh"] Oct 31 23:11:05.376611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.376533 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:11:05.376611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.376585 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:11:05.377735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.377719 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.494703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.494653 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:11:05.578612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.578570 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6wj\" (UniqueName: \"kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.578855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.578836 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.579227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.579209 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.692572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.691875 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6wj\" (UniqueName: \"kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.693063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.693036 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.693387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.693365 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.697736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.697711 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.698482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.698461 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.732129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.730910 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:11:05.732695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.732655 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:11:05.734421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.734392 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:11:05.742088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.742060 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6wj\" (UniqueName: \"kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj\") pod \"maas-api-revoke-sub-fbcadf2c-26cgh\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.879903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.879638 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:05.922793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.913316 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck"] Oct 31 23:11:05.936457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:05.936359 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:06.113016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.112948 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" event={"ID":"43bfe778-b8a8-4309-b30e-03c962094983","Type":"ContainerStarted","Data":"74c79a44bc4c7486c6c450b0c3a79ffcc14680259a80a7cd961ce8cad238a0a4"} Oct 31 23:11:06.230630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.230522 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk"] Oct 31 23:11:06.245841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.238631 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.245841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.243632 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk"] Oct 31 23:11:06.285843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.285815 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh"] Oct 31 23:11:06.403118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.403087 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.403118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.403125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.403327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.403280 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlh8\" (UniqueName: \"kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.504144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.504113 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.504297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.504147 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.504297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.504230 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlh8\" (UniqueName: \"kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.504894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.504871 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.515508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.515483 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.518444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.518408 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlh8\" (UniqueName: \"kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8\") pod \"maas-api-revoke-sub-4c95ca12-4cxsk\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.567445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.567415 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:06.852925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:06.852898 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk"] Oct 31 23:11:06.853650 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:06.853621 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0262ee16_d19e_4edc_90a2_393a65efca41.slice/crio-b8b8893e2581bab511253fa18f9d09f0b92055120f031062b07bfb535cbd91a6 WatchSource:0}: Error finding container b8b8893e2581bab511253fa18f9d09f0b92055120f031062b07bfb535cbd91a6: Status 404 returned error can't find the container with id b8b8893e2581bab511253fa18f9d09f0b92055120f031062b07bfb535cbd91a6 Oct 31 23:11:07.003328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.003280 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:07.131748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.131662 5043 generic.go:356] "Generic (PLEG): container finished" podID="43bfe778-b8a8-4309-b30e-03c962094983" containerID="31f94bcd9ebb0366ebccaadeefdc5d00210beb9d4737a72cb5c5de9b38e5e5a8" exitCode=0 Oct 31 23:11:07.131908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.131746 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" event={"ID":"43bfe778-b8a8-4309-b30e-03c962094983","Type":"ContainerDied","Data":"31f94bcd9ebb0366ebccaadeefdc5d00210beb9d4737a72cb5c5de9b38e5e5a8"} Oct 31 23:11:07.133190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.133166 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" event={"ID":"0262ee16-d19e-4edc-90a2-393a65efca41","Type":"ContainerStarted","Data":"b8b8893e2581bab511253fa18f9d09f0b92055120f031062b07bfb535cbd91a6"} Oct 31 23:11:07.134770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.134739 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" event={"ID":"d28c5f7a-39f4-492d-b560-09f385366c29","Type":"ContainerStarted","Data":"9f49382b3d03a17a0f23e56ed1e147acf3676128b5770dc71921af4614c16e31"} Oct 31 23:11:07.134862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.134795 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" event={"ID":"d28c5f7a-39f4-492d-b560-09f385366c29","Type":"ContainerStarted","Data":"5418e440d4e90ea32b3715cb9bd7b4c54e4b420a9cf387d88e77fd91b05ea970"} Oct 31 23:11:07.169873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.169795 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" podStartSLOduration=3.169776384 podStartE2EDuration="3.169776384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:07.167116985 +0000 UTC m=+1293.396671396" watchObservedRunningTime="2026-10-31 23:11:07.169776384 +0000 UTC m=+1293.399330796" Oct 31 23:11:07.715662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.713499 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:07.980727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.980635 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s"] Oct 31 23:11:07.986602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.985907 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:07.991713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:07.991676 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s"] Oct 31 23:11:08.126111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.126077 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.126288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.126134 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.126288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.126204 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8lj\" (UniqueName: \"kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.145164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.145129 5043 generic.go:356] "Generic (PLEG): container finished" podID="d28c5f7a-39f4-492d-b560-09f385366c29" containerID="9f49382b3d03a17a0f23e56ed1e147acf3676128b5770dc71921af4614c16e31" exitCode=0 Oct 31 23:11:08.145319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.145174 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" event={"ID":"d28c5f7a-39f4-492d-b560-09f385366c29","Type":"ContainerDied","Data":"9f49382b3d03a17a0f23e56ed1e147acf3676128b5770dc71921af4614c16e31"} Oct 31 23:11:08.146971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.146949 5043 generic.go:356] "Generic (PLEG): container finished" podID="0262ee16-d19e-4edc-90a2-393a65efca41" containerID="5b2345c36527582316deada4b89a648b3f0c70d42d474f66d1e3783d87e4a488" exitCode=0 Oct 31 23:11:08.147060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.147036 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" event={"ID":"0262ee16-d19e-4edc-90a2-393a65efca41","Type":"ContainerDied","Data":"5b2345c36527582316deada4b89a648b3f0c70d42d474f66d1e3783d87e4a488"} Oct 31 23:11:08.227205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.227172 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.227351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.227230 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.227351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.227257 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8lj\" (UniqueName: \"kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.228048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.228021 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.229808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.229776 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.243044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.242924 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8lj\" (UniqueName: \"kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj\") pod \"maas-api-revoke-sub-c1ae13b5-56g9s\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.305237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.305209 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:08.342599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.342572 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:08.399734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.399691 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:08.532708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.532617 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m\" (UniqueName: \"kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m\") pod \"43bfe778-b8a8-4309-b30e-03c962094983\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " Oct 31 23:11:08.532708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.532676 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca\") pod \"43bfe778-b8a8-4309-b30e-03c962094983\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " Oct 31 23:11:08.532951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.532742 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca\") pod \"43bfe778-b8a8-4309-b30e-03c962094983\" (UID: \"43bfe778-b8a8-4309-b30e-03c962094983\") " Oct 31 23:11:08.533137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.533107 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca" pod "43bfe778-b8a8-4309-b30e-03c962094983" (UID: "43bfe778-b8a8-4309-b30e-03c962094983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:08.533308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.533281 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:08.534925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.534901 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca" pod "43bfe778-b8a8-4309-b30e-03c962094983" (UID: "43bfe778-b8a8-4309-b30e-03c962094983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:08.535045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.534924 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m" pod "43bfe778-b8a8-4309-b30e-03c962094983" (UID: "43bfe778-b8a8-4309-b30e-03c962094983"). InnerVolumeSpecName "kube-api-access-fw77m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:08.588679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.588649 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s"] Oct 31 23:11:08.589045 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:08.589019 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20deaa91_4004_4e33_be85_a3cb7279b160.slice/crio-c9ac82323d845e01e8bf36c4986b58f475391347b6bb2c4fba8898c9efc20ac6 WatchSource:0}: Error finding container c9ac82323d845e01e8bf36c4986b58f475391347b6bb2c4fba8898c9efc20ac6: Status 404 returned error can't find the container with id c9ac82323d845e01e8bf36c4986b58f475391347b6bb2c4fba8898c9efc20ac6 Oct 31 23:11:08.635022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.634994 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw77m\" (UniqueName: \"kubernetes.io/projected/43bfe778-b8a8-4309-b30e-03c962094983-kube-api-access-fw77m\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:08.635022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.635021 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bfe778-b8a8-4309-b30e-03c962094983-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:08.944015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:08.943963 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:09.101414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.101380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:09.135135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.135090 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd"] Oct 31 23:11:09.135966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.135934 5043 cpu_manager.go:414] "containerMap: removing container" podUID="43bfe778-b8a8-4309-b30e-03c962094983" containerName="revoke-keys" Oct 31 23:11:09.135966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.135956 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bfe778-b8a8-4309-b30e-03c962094983" containerName="revoke-keys" Oct 31 23:11:09.136167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.136079 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bfe778-b8a8-4309-b30e-03c962094983" containerName="revoke-keys" Oct 31 23:11:09.145008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.144969 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.160209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.159997 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd"] Oct 31 23:11:09.160209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.160065 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" event={"ID":"43bfe778-b8a8-4309-b30e-03c962094983","Type":"ContainerDied","Data":"74c79a44bc4c7486c6c450b0c3a79ffcc14680259a80a7cd961ce8cad238a0a4"} Oct 31 23:11:09.160209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.160100 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c79a44bc4c7486c6c450b0c3a79ffcc14680259a80a7cd961ce8cad238a0a4" Oct 31 23:11:09.160209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.160170 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck" Oct 31 23:11:09.173898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.173850 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" event={"ID":"20deaa91-4004-4e33-be85-a3cb7279b160","Type":"ContainerStarted","Data":"4246a31c9e64a1776d2e9c2d24fdac728521f241b1f7a291fef282e2de16024c"} Oct 31 23:11:09.174027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.173917 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" event={"ID":"20deaa91-4004-4e33-be85-a3cb7279b160","Type":"ContainerStarted","Data":"c9ac82323d845e01e8bf36c4986b58f475391347b6bb2c4fba8898c9efc20ac6"} Oct 31 23:11:09.244459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.244372 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.244613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.244518 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.244841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.244821 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxrw\" (UniqueName: \"kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.346023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.345986 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxrw\" (UniqueName: \"kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.346023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.346037 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.346332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.346078 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.347177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.347153 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.348733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.348708 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.384520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.384480 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxrw\" (UniqueName: \"kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw\") pod \"maas-api-revoke-sub-75ace810-blzdd\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.420489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.420436 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" podStartSLOduration=2.420417639 podStartE2EDuration="2.420417639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:09.205967407 +0000 UTC m=+1295.435521814" watchObservedRunningTime="2026-10-31 23:11:09.420417639 +0000 UTC m=+1295.649972053" Oct 31 23:11:09.422820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.422793 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4"] Oct 31 23:11:09.423446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.423420 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:09.423904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.423884 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0262ee16-d19e-4edc-90a2-393a65efca41" containerName="revoke-keys" Oct 31 23:11:09.424112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.424063 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0262ee16-d19e-4edc-90a2-393a65efca41" containerName="revoke-keys" Oct 31 23:11:09.424320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.424298 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0262ee16-d19e-4edc-90a2-393a65efca41" containerName="revoke-keys" Oct 31 23:11:09.434798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.434766 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:09.434925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.434841 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.474874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.474842 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:09.486321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.486282 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4"] Oct 31 23:11:09.509373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.509271 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb"] Oct 31 23:11:09.510157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.510129 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d28c5f7a-39f4-492d-b560-09f385366c29" containerName="revoke-keys" Oct 31 23:11:09.510272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.510157 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28c5f7a-39f4-492d-b560-09f385366c29" containerName="revoke-keys" Oct 31 23:11:09.510360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.510344 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28c5f7a-39f4-492d-b560-09f385366c29" containerName="revoke-keys" Oct 31 23:11:09.516468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.516444 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.544323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.540956 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb"] Oct 31 23:11:09.550609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.547781 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj\" (UniqueName: \"kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj\") pod \"d28c5f7a-39f4-492d-b560-09f385366c29\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " Oct 31 23:11:09.551183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.551154 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca\") pod \"d28c5f7a-39f4-492d-b560-09f385366c29\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " Oct 31 23:11:09.551682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.551657 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca\") pod \"0262ee16-d19e-4edc-90a2-393a65efca41\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " Oct 31 23:11:09.551753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.551703 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8\" (UniqueName: \"kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8\") pod \"0262ee16-d19e-4edc-90a2-393a65efca41\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " Oct 31 23:11:09.551806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.551757 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca\") pod \"d28c5f7a-39f4-492d-b560-09f385366c29\" (UID: \"d28c5f7a-39f4-492d-b560-09f385366c29\") " Oct 31 23:11:09.551915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.551898 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca\") pod \"0262ee16-d19e-4edc-90a2-393a65efca41\" (UID: \"0262ee16-d19e-4edc-90a2-393a65efca41\") " Oct 31 23:11:09.552095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552074 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74cc\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552129 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552156 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552295 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552336 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552376 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552423 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9cc\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552499 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552534 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552599 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552631 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552687 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552715 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552770 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552795 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.552950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552824 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.552950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552909 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.553098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.552974 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.553602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.553564 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca" pod "d28c5f7a-39f4-492d-b560-09f385366c29" (UID: "d28c5f7a-39f4-492d-b560-09f385366c29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:09.553797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.553776 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca" pod "0262ee16-d19e-4edc-90a2-393a65efca41" (UID: "0262ee16-d19e-4edc-90a2-393a65efca41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:09.620211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.614106 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca" pod "0262ee16-d19e-4edc-90a2-393a65efca41" (UID: "0262ee16-d19e-4edc-90a2-393a65efca41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:09.620211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.614140 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj" pod "d28c5f7a-39f4-492d-b560-09f385366c29" (UID: "d28c5f7a-39f4-492d-b560-09f385366c29"). InnerVolumeSpecName "kube-api-access-fs6wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:09.620211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.614202 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca" pod "d28c5f7a-39f4-492d-b560-09f385366c29" (UID: "d28c5f7a-39f4-492d-b560-09f385366c29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:09.620211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.614685 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8" pod "0262ee16-d19e-4edc-90a2-393a65efca41" (UID: "0262ee16-d19e-4edc-90a2-393a65efca41"). InnerVolumeSpecName "kube-api-access-6xlh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:09.654255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654216 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.654487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654284 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.654487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654317 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.654731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654710 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.654838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654748 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.654838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654799 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.654838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654825 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.654986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654855 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.654986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654930 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.654993 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655029 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74cc\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.655100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655060 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655067 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.655100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655085 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.655339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655165 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655191 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.655339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655227 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655273 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9cc\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655541 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655356 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.655541 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655385 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.655790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655697 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.655928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655827 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:09.656000 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.655945 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656104 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656304 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656365 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656562 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656611 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656672 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6wj\" (UniqueName: \"kubernetes.io/projected/d28c5f7a-39f4-492d-b560-09f385366c29-kube-api-access-fs6wj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656704 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656723 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262ee16-d19e-4edc-90a2-393a65efca41-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656741 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlh8\" (UniqueName: \"kubernetes.io/projected/0262ee16-d19e-4edc-90a2-393a65efca41-kube-api-access-6xlh8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:09.656825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656759 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28c5f7a-39f4-492d-b560-09f385366c29-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:09.657414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.656856 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.658292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.658269 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.658362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.658347 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.659443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.659421 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.660151 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.660131 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.668747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.668716 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9cc\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.668921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.668882 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.671517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.671458 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74cc\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.674761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.674733 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.761033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.760940 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:09.847406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.847162 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:09.857683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:09.857253 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd"] Oct 31 23:11:09.868840 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:09.865110 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d81d6c_ce05_4db7_a3dd_3724fd75a874.slice/crio-b6483450c0703a1a9004ae571280cabb9d21014413bb6fba21d7a8a64731f16a WatchSource:0}: Error finding container b6483450c0703a1a9004ae571280cabb9d21014413bb6fba21d7a8a64731f16a: Status 404 returned error can't find the container with id b6483450c0703a1a9004ae571280cabb9d21014413bb6fba21d7a8a64731f16a Oct 31 23:11:10.070512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.070482 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4"] Oct 31 23:11:10.070846 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:10.070806 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81afb95_ac07_4546_bea2_dfc33caf888d.slice/crio-a4a8a4432216e13a6777d53b99bcd8759fdcdf333abe994ae8c94f5fa1a8b875 WatchSource:0}: Error finding container a4a8a4432216e13a6777d53b99bcd8759fdcdf333abe994ae8c94f5fa1a8b875: Status 404 returned error can't find the container with id a4a8a4432216e13a6777d53b99bcd8759fdcdf333abe994ae8c94f5fa1a8b875 Oct 31 23:11:10.073265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.073235 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:10.073373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.073295 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:10.073373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.073325 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:10.119608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.115526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:10.171165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.171143 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb"] Oct 31 23:11:10.173691 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:10.173662 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eab6b9_e331_48da_bfbf_0d04e0baf215.slice/crio-7ae729751b5945f459e252110f10de415a2bb7a3bc19080f4fcabc4b3feaa909 WatchSource:0}: Error finding container 7ae729751b5945f459e252110f10de415a2bb7a3bc19080f4fcabc4b3feaa909: Status 404 returned error can't find the container with id 7ae729751b5945f459e252110f10de415a2bb7a3bc19080f4fcabc4b3feaa909 Oct 31 23:11:10.176857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.176796 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:10.176984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.176879 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:10.176984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.176908 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:10.205286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.205246 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" event={"ID":"0262ee16-d19e-4edc-90a2-393a65efca41","Type":"ContainerDied","Data":"b8b8893e2581bab511253fa18f9d09f0b92055120f031062b07bfb535cbd91a6"} Oct 31 23:11:10.205611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.205587 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8b8893e2581bab511253fa18f9d09f0b92055120f031062b07bfb535cbd91a6" Oct 31 23:11:10.206025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.205731 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk" Oct 31 23:11:10.208350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.208323 5043 generic.go:356] "Generic (PLEG): container finished" podID="20deaa91-4004-4e33-be85-a3cb7279b160" containerID="4246a31c9e64a1776d2e9c2d24fdac728521f241b1f7a291fef282e2de16024c" exitCode=0 Oct 31 23:11:10.208466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.208449 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" event={"ID":"20deaa91-4004-4e33-be85-a3cb7279b160","Type":"ContainerDied","Data":"4246a31c9e64a1776d2e9c2d24fdac728521f241b1f7a291fef282e2de16024c"} Oct 31 23:11:10.210218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.210196 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" event={"ID":"63d81d6c-ce05-4db7-a3dd-3724fd75a874","Type":"ContainerStarted","Data":"b6483450c0703a1a9004ae571280cabb9d21014413bb6fba21d7a8a64731f16a"} Oct 31 23:11:10.212297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.212271 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" event={"ID":"33eab6b9-e331-48da-bfbf-0d04e0baf215","Type":"ContainerStarted","Data":"7ae729751b5945f459e252110f10de415a2bb7a3bc19080f4fcabc4b3feaa909"} Oct 31 23:11:10.214392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.214364 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" event={"ID":"f81afb95-ac07-4546-bea2-dfc33caf888d","Type":"ContainerStarted","Data":"c80e9eb07b05228715ce9d5fb8f1ea4fc46c76023c0c99f5b2867ca1725362a9"} Oct 31 23:11:10.214509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.214408 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" event={"ID":"f81afb95-ac07-4546-bea2-dfc33caf888d","Type":"ContainerStarted","Data":"a4a8a4432216e13a6777d53b99bcd8759fdcdf333abe994ae8c94f5fa1a8b875"} Oct 31 23:11:10.218193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.218167 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" event={"ID":"d28c5f7a-39f4-492d-b560-09f385366c29","Type":"ContainerDied","Data":"5418e440d4e90ea32b3715cb9bd7b4c54e4b420a9cf387d88e77fd91b05ea970"} Oct 31 23:11:10.218306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.218218 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5418e440d4e90ea32b3715cb9bd7b4c54e4b420a9cf387d88e77fd91b05ea970" Oct 31 23:11:10.218306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.218293 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh" Oct 31 23:11:10.263567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.263437 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podStartSLOduration=1.263416656 podStartE2EDuration="1.263416656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:10.26018097 +0000 UTC m=+1296.489735382" watchObservedRunningTime="2026-10-31 23:11:10.263416656 +0000 UTC m=+1296.492971068" Oct 31 23:11:10.460777 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:11:10.460736 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:11:10.764603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:10.761766 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:11.040581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.040449 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:11.263653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.263613 5043 generic.go:356] "Generic (PLEG): container finished" podID="63d81d6c-ce05-4db7-a3dd-3724fd75a874" containerID="cab6cbe3af34a613660a08f3c4dfc2c994ea3dfdec42c54535622db8b58e4cfc" exitCode=0 Oct 31 23:11:11.264292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.264264 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" event={"ID":"63d81d6c-ce05-4db7-a3dd-3724fd75a874","Type":"ContainerDied","Data":"cab6cbe3af34a613660a08f3c4dfc2c994ea3dfdec42c54535622db8b58e4cfc"} Oct 31 23:11:11.274531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.274499 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" event={"ID":"33eab6b9-e331-48da-bfbf-0d04e0baf215","Type":"ContainerStarted","Data":"4854b34e74d727bf0de49827be4ab51521d669dc74f5a4b16eaee49d3b7f1376"} Oct 31 23:11:11.341324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.341202 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podStartSLOduration=2.341180961 podStartE2EDuration="2.341180961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:11.322391961 +0000 UTC m=+1297.551946384" watchObservedRunningTime="2026-10-31 23:11:11.341180961 +0000 UTC m=+1297.570735372" Oct 31 23:11:11.612911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.612887 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:11.695965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.695926 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca\") pod \"20deaa91-4004-4e33-be85-a3cb7279b160\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " Oct 31 23:11:11.696632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.696608 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca\") pod \"20deaa91-4004-4e33-be85-a3cb7279b160\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " Oct 31 23:11:11.697183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.697006 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca" pod "20deaa91-4004-4e33-be85-a3cb7279b160" (UID: "20deaa91-4004-4e33-be85-a3cb7279b160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:11.697343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.697283 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj\" (UniqueName: \"kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj\") pod \"20deaa91-4004-4e33-be85-a3cb7279b160\" (UID: \"20deaa91-4004-4e33-be85-a3cb7279b160\") " Oct 31 23:11:11.698099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.698077 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:11.698476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.698454 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca" pod "20deaa91-4004-4e33-be85-a3cb7279b160" (UID: "20deaa91-4004-4e33-be85-a3cb7279b160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:11.698872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.698842 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj" pod "20deaa91-4004-4e33-be85-a3cb7279b160" (UID: "20deaa91-4004-4e33-be85-a3cb7279b160"). InnerVolumeSpecName "kube-api-access-nj8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:11.770008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.769660 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:11:11.770008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.769724 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 23:11:11.799322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.799271 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20deaa91-4004-4e33-be85-a3cb7279b160-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:11.799322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.799300 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8lj\" (UniqueName: \"kubernetes.io/projected/20deaa91-4004-4e33-be85-a3cb7279b160-kube-api-access-nj8lj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:11.849998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.849864 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:11.853967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.853648 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:11:11.853967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:11.853704 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 23:11:12.185719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.185622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:12.283475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.283440 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" Oct 31 23:11:12.283880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.283851 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s" event={"ID":"20deaa91-4004-4e33-be85-a3cb7279b160","Type":"ContainerDied","Data":"c9ac82323d845e01e8bf36c4986b58f475391347b6bb2c4fba8898c9efc20ac6"} Oct 31 23:11:12.283925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.283884 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ac82323d845e01e8bf36c4986b58f475391347b6bb2c4fba8898c9efc20ac6" Oct 31 23:11:12.578704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.578670 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:12.611057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.611033 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca\") pod \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " Oct 31 23:11:12.611634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.611610 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca\") pod \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " Oct 31 23:11:12.611769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.611658 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw\" (UniqueName: \"kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw\") pod \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\" (UID: \"63d81d6c-ce05-4db7-a3dd-3724fd75a874\") " Oct 31 23:11:12.612222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.612180 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca" pod "63d81d6c-ce05-4db7-a3dd-3724fd75a874" (UID: "63d81d6c-ce05-4db7-a3dd-3724fd75a874"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:12.663264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.662895 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca" pod "63d81d6c-ce05-4db7-a3dd-3724fd75a874" (UID: "63d81d6c-ce05-4db7-a3dd-3724fd75a874"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:12.663264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.663052 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw" pod "63d81d6c-ce05-4db7-a3dd-3724fd75a874" (UID: "63d81d6c-ce05-4db7-a3dd-3724fd75a874"). InnerVolumeSpecName "kube-api-access-mdxrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:12.666489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.663808 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxrw\" (UniqueName: \"kubernetes.io/projected/63d81d6c-ce05-4db7-a3dd-3724fd75a874-kube-api-access-mdxrw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:12.666489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.663848 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:12.666489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.663866 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d81d6c-ce05-4db7-a3dd-3724fd75a874-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:12.762005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.761964 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:11:12.762269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.762240 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 23:11:12.850050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.849953 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:11:12.850317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:12.850287 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 23:11:13.184157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.184050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:13.293833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.293802 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" Oct 31 23:11:13.294181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.293874 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd" event={"ID":"63d81d6c-ce05-4db7-a3dd-3724fd75a874","Type":"ContainerDied","Data":"b6483450c0703a1a9004ae571280cabb9d21014413bb6fba21d7a8a64731f16a"} Oct 31 23:11:13.294181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.293928 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6483450c0703a1a9004ae571280cabb9d21014413bb6fba21d7a8a64731f16a" Oct 31 23:11:13.763396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.762952 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:11:13.763396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.763015 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 23:11:13.852740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.852707 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:11:13.853001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:13.852977 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 23:11:14.255574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:14.251121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:14.764047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:14.764014 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:11:14.764642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:14.764597 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 23:11:14.853571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:14.851947 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:11:14.853571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:14.852012 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 23:11:15.376572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:15.373969 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:15.586594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:15.584117 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:15.762303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:15.762261 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:11:15.762608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:15.762577 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 23:11:15.850571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:15.849984 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:11:15.850571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:15.850043 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 23:11:16.463784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:16.463720 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:16.762055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:16.762013 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 23:11:16.762248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:16.762085 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 23:11:16.849089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:16.849050 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 23:11:16.849268 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:16.849117 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 23:11:17.329068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.329036 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:17.765878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.765845 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:17.766156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.766129 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:17.766970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.766948 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:11:17.853682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.853648 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:17.854163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.854140 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:17.855017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:17.854998 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:11:18.192805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.192711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:18.367927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.367885 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf"] Oct 31 23:11:18.368637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.368615 5043 cpu_manager.go:414] "containerMap: removing container" podUID="20deaa91-4004-4e33-be85-a3cb7279b160" containerName="revoke-keys" Oct 31 23:11:18.368637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.368637 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20deaa91-4004-4e33-be85-a3cb7279b160" containerName="revoke-keys" Oct 31 23:11:18.368794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.368689 5043 cpu_manager.go:414] "containerMap: removing container" podUID="63d81d6c-ce05-4db7-a3dd-3724fd75a874" containerName="revoke-keys" Oct 31 23:11:18.368794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.368697 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d81d6c-ce05-4db7-a3dd-3724fd75a874" containerName="revoke-keys" Oct 31 23:11:18.368899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.368821 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="20deaa91-4004-4e33-be85-a3cb7279b160" containerName="revoke-keys" Oct 31 23:11:18.368899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.368845 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d81d6c-ce05-4db7-a3dd-3724fd75a874" containerName="revoke-keys" Oct 31 23:11:18.374589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.374535 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.380105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.380078 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf"] Oct 31 23:11:18.381454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.381360 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:11:18.381454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.381359 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:11:18.381708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.381611 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:11:18.381708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.381634 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:11:18.434802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.434764 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.435005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.434812 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.435005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.434985 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89qw\" (UniqueName: \"kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.535974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.535929 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89qw\" (UniqueName: \"kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.536273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.536237 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.536430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.536301 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.537074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.537055 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.538680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.538662 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.555445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.555413 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89qw\" (UniqueName: \"kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw\") pod \"maas-api-revoke-sub-ee75fd08-dgrlf\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:18.708569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:18.701737 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:20.020573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.012234 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:20.181663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.159404 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf"] Oct 31 23:11:20.258583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.240817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:20.297631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.283810 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:20.331568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.317761 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:11:20.387364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.387321 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:20.622791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.622684 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:20.781141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.781027 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" event={"ID":"a8a237a2-25ab-43b9-a748-359a585a6b0e","Type":"ContainerStarted","Data":"250fdb57b5fb3f08cb7e18b9a356f96cbd60ceb3b5d3cd7a89be8fd9be0186e7"} Oct 31 23:11:20.885891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:20.884676 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:21.031863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.026899 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:11:21.304569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.261220 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:21.304569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.261532 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 31 23:11:21.342616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.333653 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:21.342616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.334466 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:11:21.481269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.481222 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:21.482232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.482211 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:11:21.503697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.503642 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:21.552572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.516069 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:11:21.638942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.638831 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:21.653703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.653664 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:11:21.924919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.916221 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:21.954585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:21.925317 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:11:22.118701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.118652 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:11:22.178343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.147318 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:22.337567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.327947 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:22.337567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.336085 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:22.479717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.479631 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" event={"ID":"a8a237a2-25ab-43b9-a748-359a585a6b0e","Type":"ContainerStarted","Data":"7faac352079ca1db03a2e6e17ecd52ae735908e138875f75a2c4d0318e3b35e4"} Oct 31 23:11:22.480026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.480006 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:11:22.480478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.480461 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:11:22.481365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.481350 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:11:22.482330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.482314 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:11:22.482509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.482495 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:11:22.483206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.483190 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:11:22.498997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.498964 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8"] Oct 31 23:11:22.801175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.801146 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:11:22.801519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.801501 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8"] Oct 31 23:11:22.801772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.801453 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:22.808246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.808195 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" podStartSLOduration=4.808178414 podStartE2EDuration="4.808178414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:22.21793691 +0000 UTC m=+1308.447491324" watchObservedRunningTime="2026-10-31 23:11:22.808178414 +0000 UTC m=+1309.037732826" Oct 31 23:11:22.820746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.819918 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-67fce9\"/\"distinct-w3-67fce9-kserve-self-signed-certs\"" Oct 31 23:11:22.977571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:22.974928 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:23.006450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.005948 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.006450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.005996 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.006450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.006091 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb4p\" (UniqueName: \"kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.006450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.006160 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.006450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.006208 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.006450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.006275 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.142276 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.142337 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.142445 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb4p\" (UniqueName: \"kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.142528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.142603 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.158489 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.159115 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.159164 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.169406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.159613 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.213642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.208385 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.313230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.313195 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.416571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.413520 5043 generic.go:356] "Generic (PLEG): container finished" podID="a8a237a2-25ab-43b9-a748-359a585a6b0e" containerID="7faac352079ca1db03a2e6e17ecd52ae735908e138875f75a2c4d0318e3b35e4" exitCode=0 Oct 31 23:11:23.433900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.414527 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" event={"ID":"a8a237a2-25ab-43b9-a748-359a585a6b0e","Type":"ContainerDied","Data":"7faac352079ca1db03a2e6e17ecd52ae735908e138875f75a2c4d0318e3b35e4"} Oct 31 23:11:23.433900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.415651 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb4p\" (UniqueName: \"kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p\") pod \"distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:23.642585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.610215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:23.727144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:23.727060 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:24.125080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.125039 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8"] Oct 31 23:11:24.155897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.151049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:24.282410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.282376 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx"] Oct 31 23:11:24.286961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.286935 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.298702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.298667 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx"] Oct 31 23:11:24.300837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.300791 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:24.377004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.376918 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.376972 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377000 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377104 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqxm6\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377141 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377201 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377260 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377284 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.377369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.377316 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.427485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.427444 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" event={"ID":"985fa972-caf5-46b0-be61-ae1fabb59bfe","Type":"ContainerStarted","Data":"176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc"} Oct 31 23:11:24.427720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.427494 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" event={"ID":"985fa972-caf5-46b0-be61-ae1fabb59bfe","Type":"ContainerStarted","Data":"125a33e7d3b3898cdc07abca07707397dbc1e1e210ec7ac73f6624416e93475f"} Oct 31 23:11:24.456975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.456895 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podStartSLOduration=3.456872722 podStartE2EDuration="3.456872722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:24.452444103 +0000 UTC m=+1310.681998519" watchObservedRunningTime="2026-10-31 23:11:24.456872722 +0000 UTC m=+1310.686427133" Oct 31 23:11:24.478367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.478321 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.479238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.479214 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.480466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.480447 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.481577 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.481653 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqxm6\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.481683 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.481763 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.481812 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.481848 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.482178 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.479052 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.482900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.480850 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.483410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.482920 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.483410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.480203 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.485512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.485448 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.486033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.486011 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.495327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.495301 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.495809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.495789 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqxm6\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.603078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.603039 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:24.671215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.671188 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:24.685111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.685073 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca\") pod \"a8a237a2-25ab-43b9-a748-359a585a6b0e\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " Oct 31 23:11:24.685337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.685317 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca\") pod \"a8a237a2-25ab-43b9-a748-359a585a6b0e\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " Oct 31 23:11:24.685424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.685375 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw\" (UniqueName: \"kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw\") pod \"a8a237a2-25ab-43b9-a748-359a585a6b0e\" (UID: \"a8a237a2-25ab-43b9-a748-359a585a6b0e\") " Oct 31 23:11:24.685533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.685504 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca" pod "a8a237a2-25ab-43b9-a748-359a585a6b0e" (UID: "a8a237a2-25ab-43b9-a748-359a585a6b0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:24.685923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.685900 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:24.688326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.688298 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw" pod "a8a237a2-25ab-43b9-a748-359a585a6b0e" (UID: "a8a237a2-25ab-43b9-a748-359a585a6b0e"). InnerVolumeSpecName "kube-api-access-g89qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:24.688445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.688383 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca" pod "a8a237a2-25ab-43b9-a748-359a585a6b0e" (UID: "a8a237a2-25ab-43b9-a748-359a585a6b0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:24.744957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.744825 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:24.787584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.787527 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a237a2-25ab-43b9-a748-359a585a6b0e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:24.787584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.787584 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89qw\" (UniqueName: \"kubernetes.io/projected/a8a237a2-25ab-43b9-a748-359a585a6b0e-kube-api-access-g89qw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:24.914018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.913938 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:24.928484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.928450 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx"] Oct 31 23:11:24.952739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.952700 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:24.952929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.952834 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:24.952929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:24.952891 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:25.449677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.449631 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" event={"ID":"a8a237a2-25ab-43b9-a748-359a585a6b0e","Type":"ContainerDied","Data":"250fdb57b5fb3f08cb7e18b9a356f96cbd60ceb3b5d3cd7a89be8fd9be0186e7"} Oct 31 23:11:25.450108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.449700 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="250fdb57b5fb3f08cb7e18b9a356f96cbd60ceb3b5d3cd7a89be8fd9be0186e7" Oct 31 23:11:25.450108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.449773 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf" Oct 31 23:11:25.455011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.454866 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" event={"ID":"e3f3e249-1514-4c28-a06f-d7db747a4ef9","Type":"ContainerStarted","Data":"31b604bd4cc786af918d2adbe3afc592bc2636bad0bd2e63c399d07d4578c69d"} Oct 31 23:11:25.455011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.454926 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" event={"ID":"e3f3e249-1514-4c28-a06f-d7db747a4ef9","Type":"ContainerStarted","Data":"611b0be09330c24d5463b4ff973a576aceb3ffb71502e993552a95818920699a"} Oct 31 23:11:25.510221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.510176 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podStartSLOduration=1.5101618559999999 podStartE2EDuration="1.510161856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:25.507841241 +0000 UTC m=+1311.737395679" watchObservedRunningTime="2026-10-31 23:11:25.510161856 +0000 UTC m=+1311.739716267" Oct 31 23:11:25.603632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.603598 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:25.901064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:25.901030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:26.605416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:26.605375 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:11:26.605809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:26.605453 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 23:11:26.647568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:26.640127 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:27.337646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.337601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:27.603568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.603471 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:11:27.603568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.603523 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 23:11:27.740730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.740667 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8"] Oct 31 23:11:27.743654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.743628 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a8a237a2-25ab-43b9-a748-359a585a6b0e" containerName="revoke-keys" Oct 31 23:11:27.743654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.743653 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a237a2-25ab-43b9-a748-359a585a6b0e" containerName="revoke-keys" Oct 31 23:11:27.743898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.743882 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a237a2-25ab-43b9-a748-359a585a6b0e" containerName="revoke-keys" Oct 31 23:11:27.748247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.748223 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8"] Oct 31 23:11:27.748374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.748336 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.755748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.755714 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:11:27.756683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.756516 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:11:27.756683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.756586 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:11:27.756683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.756638 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:11:27.757527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.757360 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88"] Oct 31 23:11:27.761471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.761452 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.771947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.771922 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88"] Oct 31 23:11:27.860272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.860113 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.860272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.860190 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm9k\" (UniqueName: \"kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.860509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.860308 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.860615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.860503 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.860615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.860601 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.860765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.860678 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8xd\" (UniqueName: \"kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.961585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.961528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8xd\" (UniqueName: \"kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.961780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.961642 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.961780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.961686 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm9k\" (UniqueName: \"kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.961780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.961729 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.961946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.961847 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.961946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.961909 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.962484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.962460 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.962712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.962681 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.964832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.964809 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:27.964832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.964821 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.973896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.973867 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm9k\" (UniqueName: \"kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k\") pod \"maas-api-revoke-sub-f0497054-hlb88\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:27.975004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:27.974982 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8xd\" (UniqueName: \"kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd\") pod \"maas-api-revoke-sub-114db158-8qlj8\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:28.066274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.066232 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:28.079068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.079044 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:28.147150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.146990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:28.412579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.412359 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8"] Oct 31 23:11:28.412822 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:28.412799 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbdbff0_b677_4495_95a6_f5884856bc4a.slice/crio-6adf091d5f21ef80fa65d7a138a5b4554a3594b6e385f07266da8bc88a0d72cf WatchSource:0}: Error finding container 6adf091d5f21ef80fa65d7a138a5b4554a3594b6e385f07266da8bc88a0d72cf: Status 404 returned error can't find the container with id 6adf091d5f21ef80fa65d7a138a5b4554a3594b6e385f07266da8bc88a0d72cf Oct 31 23:11:28.494184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.494143 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" event={"ID":"2fbdbff0-b677-4495-95a6-f5884856bc4a","Type":"ContainerStarted","Data":"6adf091d5f21ef80fa65d7a138a5b4554a3594b6e385f07266da8bc88a0d72cf"} Oct 31 23:11:28.495490 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:28.495451 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27be5640_28a4_4950_856a_628e861d09fb.slice/crio-2a4d38c1d3937bc6a38474225e995260887f4facf9d1a39eac31cf2a8adfc390 WatchSource:0}: Error finding container 2a4d38c1d3937bc6a38474225e995260887f4facf9d1a39eac31cf2a8adfc390: Status 404 returned error can't find the container with id 2a4d38c1d3937bc6a38474225e995260887f4facf9d1a39eac31cf2a8adfc390 Oct 31 23:11:28.496123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.496095 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88"] Oct 31 23:11:28.604192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.604158 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:11:28.604372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.604223 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 23:11:28.959962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:28.959929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:29.503926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.503866 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" event={"ID":"2fbdbff0-b677-4495-95a6-f5884856bc4a","Type":"ContainerStarted","Data":"9531081ba2dde1c7f91be9432dace3a856b98affa0ffa0b2f24033cd16c9f644"} Oct 31 23:11:29.505601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.505574 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" event={"ID":"27be5640-28a4-4950-856a-628e861d09fb","Type":"ContainerStarted","Data":"950e4e3f1b5bc69764dbb29b01483fb7d5a93dc309c844b9260dd23cdcd1ce7a"} Oct 31 23:11:29.505728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.505622 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" event={"ID":"27be5640-28a4-4950-856a-628e861d09fb","Type":"ContainerStarted","Data":"2a4d38c1d3937bc6a38474225e995260887f4facf9d1a39eac31cf2a8adfc390"} Oct 31 23:11:29.547061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.546997 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" podStartSLOduration=2.546978805 podStartE2EDuration="2.546978805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:29.545624642 +0000 UTC m=+1315.775179053" watchObservedRunningTime="2026-10-31 23:11:29.546978805 +0000 UTC m=+1315.776533216" Oct 31 23:11:29.570300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.570240 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" podStartSLOduration=2.570224855 podStartE2EDuration="2.570224855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:29.56996788 +0000 UTC m=+1315.799522290" watchObservedRunningTime="2026-10-31 23:11:29.570224855 +0000 UTC m=+1315.799779265" Oct 31 23:11:29.604092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.604052 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:11:29.604259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.604114 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 23:11:29.644012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:29.640142 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:30.438578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.434764 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:30.514535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.514494 5043 generic.go:356] "Generic (PLEG): container finished" podID="2fbdbff0-b677-4495-95a6-f5884856bc4a" containerID="9531081ba2dde1c7f91be9432dace3a856b98affa0ffa0b2f24033cd16c9f644" exitCode=0 Oct 31 23:11:30.514904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.514877 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" event={"ID":"2fbdbff0-b677-4495-95a6-f5884856bc4a","Type":"ContainerDied","Data":"9531081ba2dde1c7f91be9432dace3a856b98affa0ffa0b2f24033cd16c9f644"} Oct 31 23:11:30.527735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.527698 5043 generic.go:356] "Generic (PLEG): container finished" podID="27be5640-28a4-4950-856a-628e861d09fb" containerID="950e4e3f1b5bc69764dbb29b01483fb7d5a93dc309c844b9260dd23cdcd1ce7a" exitCode=0 Oct 31 23:11:30.529302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.528073 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" event={"ID":"27be5640-28a4-4950-856a-628e861d09fb","Type":"ContainerDied","Data":"950e4e3f1b5bc69764dbb29b01483fb7d5a93dc309c844b9260dd23cdcd1ce7a"} Oct 31 23:11:30.604156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.604116 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 23:11:30.604320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:30.604205 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 23:11:31.141287 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.141245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:31.738780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.735127 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:31.738780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.736191 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:31.746654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.742437 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:11:31.755507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.755476 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:31.800818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.800740 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:31.802430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.802409 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:31.817234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.816168 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd\" (UniqueName: \"kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd\") pod \"2fbdbff0-b677-4495-95a6-f5884856bc4a\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " Oct 31 23:11:31.817234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.816315 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca\") pod \"2fbdbff0-b677-4495-95a6-f5884856bc4a\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " Oct 31 23:11:31.817234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.816397 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca\") pod \"2fbdbff0-b677-4495-95a6-f5884856bc4a\" (UID: \"2fbdbff0-b677-4495-95a6-f5884856bc4a\") " Oct 31 23:11:31.839569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.820633 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca" pod "2fbdbff0-b677-4495-95a6-f5884856bc4a" (UID: "2fbdbff0-b677-4495-95a6-f5884856bc4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:31.839569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.831030 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd" pod "2fbdbff0-b677-4495-95a6-f5884856bc4a" (UID: "2fbdbff0-b677-4495-95a6-f5884856bc4a"). InnerVolumeSpecName "kube-api-access-cs8xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:31.851596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.841131 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca" pod "2fbdbff0-b677-4495-95a6-f5884856bc4a" (UID: "2fbdbff0-b677-4495-95a6-f5884856bc4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:31.918891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.918859 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca\") pod \"27be5640-28a4-4950-856a-628e861d09fb\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " Oct 31 23:11:31.919083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.918901 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k\" (UniqueName: \"kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k\") pod \"27be5640-28a4-4950-856a-628e861d09fb\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " Oct 31 23:11:31.919083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.918945 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca\") pod \"27be5640-28a4-4950-856a-628e861d09fb\" (UID: \"27be5640-28a4-4950-856a-628e861d09fb\") " Oct 31 23:11:31.929788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.929660 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8xd\" (UniqueName: \"kubernetes.io/projected/2fbdbff0-b677-4495-95a6-f5884856bc4a-kube-api-access-cs8xd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:31.929788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.929702 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:31.929788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.929717 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbdbff0-b677-4495-95a6-f5884856bc4a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:31.937350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.934679 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca" pod "27be5640-28a4-4950-856a-628e861d09fb" (UID: "27be5640-28a4-4950-856a-628e861d09fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:31.945354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.940303 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k" pod "27be5640-28a4-4950-856a-628e861d09fb" (UID: "27be5640-28a4-4950-856a-628e861d09fb"). InnerVolumeSpecName "kube-api-access-fdm9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:31.945354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:31.942112 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca" pod "27be5640-28a4-4950-856a-628e861d09fb" (UID: "27be5640-28a4-4950-856a-628e861d09fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:32.031106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.031054 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27be5640-28a4-4950-856a-628e861d09fb-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:32.031106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.031097 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdm9k\" (UniqueName: \"kubernetes.io/projected/27be5640-28a4-4950-856a-628e861d09fb-kube-api-access-fdm9k\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:32.031106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.031114 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27be5640-28a4-4950-856a-628e861d09fb-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:32.511872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.511831 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:32.550634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.550597 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" event={"ID":"27be5640-28a4-4950-856a-628e861d09fb","Type":"ContainerDied","Data":"2a4d38c1d3937bc6a38474225e995260887f4facf9d1a39eac31cf2a8adfc390"} Oct 31 23:11:32.550825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.550604 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88" Oct 31 23:11:32.550892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.550701 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4d38c1d3937bc6a38474225e995260887f4facf9d1a39eac31cf2a8adfc390" Oct 31 23:11:32.552358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.552327 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" event={"ID":"2fbdbff0-b677-4495-95a6-f5884856bc4a","Type":"ContainerDied","Data":"6adf091d5f21ef80fa65d7a138a5b4554a3594b6e385f07266da8bc88a0d72cf"} Oct 31 23:11:32.552506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.552362 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8" Oct 31 23:11:32.552506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.552381 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6adf091d5f21ef80fa65d7a138a5b4554a3594b6e385f07266da8bc88a0d72cf" Oct 31 23:11:32.666593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:32.665372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:33.219083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:33.219048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:33.744902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:33.744857 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:33.759522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:33.759483 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:33.975489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:33.975452 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:34.118670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.118512 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7"] Oct 31 23:11:34.119454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.119429 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2fbdbff0-b677-4495-95a6-f5884856bc4a" containerName="revoke-keys" Oct 31 23:11:34.119533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.119455 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbdbff0-b677-4495-95a6-f5884856bc4a" containerName="revoke-keys" Oct 31 23:11:34.119533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.119473 5043 cpu_manager.go:414] "containerMap: removing container" podUID="27be5640-28a4-4950-856a-628e861d09fb" containerName="revoke-keys" Oct 31 23:11:34.119533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.119480 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27be5640-28a4-4950-856a-628e861d09fb" containerName="revoke-keys" Oct 31 23:11:34.119743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.119702 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbdbff0-b677-4495-95a6-f5884856bc4a" containerName="revoke-keys" Oct 31 23:11:34.119743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.119718 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="27be5640-28a4-4950-856a-628e861d09fb" containerName="revoke-keys" Oct 31 23:11:34.125292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.125269 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.131591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.131535 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"distinct-w5-2294a7-kserve-self-signed-certs\"" Oct 31 23:11:34.133442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.133415 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7"] Oct 31 23:11:34.197144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.197112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:34.266061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.266026 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.266440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.266154 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.266440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.266220 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.266440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.266239 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqc6\" (UniqueName: \"kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.266440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.266305 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.266440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.266397 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.373204 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.373378 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.373456 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.373496 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqc6\" (UniqueName: \"kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.373599 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.373694 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.374521 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.374925 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.377621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.375764 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.384198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.383727 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.387293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.387257 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.388576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.388517 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqc6\" (UniqueName: \"kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6\") pod \"distinct-w5-2294a7-kserve-cc545c8fc-fdck7\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.449629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.449533 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:34.591683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.591649 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:11:34.742029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.741856 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7"] Oct 31 23:11:34.744753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:34.743662 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:34.755325 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:34.752256 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ac6cb61_cfe6_4efc_82db_95b60b80b33a.slice/crio-cf60d5e4efad889555d93ae1e6b8938191c59d853f6aff278bec48a557edd13f WatchSource:0}: Error finding container cf60d5e4efad889555d93ae1e6b8938191c59d853f6aff278bec48a557edd13f: Status 404 returned error can't find the container with id cf60d5e4efad889555d93ae1e6b8938191c59d853f6aff278bec48a557edd13f Oct 31 23:11:35.536149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:35.536112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:35.581171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:35.581130 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" event={"ID":"3ac6cb61-cfe6-4efc-82db-95b60b80b33a","Type":"ContainerStarted","Data":"4cdb91259c2612b02b50e600a37977a9094e8d130a9fae523b0b2a441a94883b"} Oct 31 23:11:35.581342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:35.581182 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" event={"ID":"3ac6cb61-cfe6-4efc-82db-95b60b80b33a","Type":"ContainerStarted","Data":"cf60d5e4efad889555d93ae1e6b8938191c59d853f6aff278bec48a557edd13f"} Oct 31 23:11:35.603524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:35.603471 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podStartSLOduration=1.603456167 podStartE2EDuration="1.603456167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:35.601172523 +0000 UTC m=+1321.830726936" watchObservedRunningTime="2026-10-31 23:11:35.603456167 +0000 UTC m=+1321.833010577" Oct 31 23:11:36.241569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:36.240469 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:36.450487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:36.450452 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:37.021570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:37.020305 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:37.869323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:37.869274 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:38.354569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.353642 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:38.355110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.354939 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:38.458570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.457673 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:38.514570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.513647 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:38.577696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.577645 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:11:38.629766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.629659 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:38.691747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.691695 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:38.985002 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:38.983640 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:39.631825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.631785 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd"] Oct 31 23:11:39.649792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.649760 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd"] Oct 31 23:11:39.650027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.649878 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.714732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.714647 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:39.728623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.728590 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll"] Oct 31 23:11:39.734286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.734253 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.773750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.773721 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll"] Oct 31 23:11:39.780035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.780008 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf"] Oct 31 23:11:39.784821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.784590 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.785466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.785308 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf"] Oct 31 23:11:39.791735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.791712 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.791959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.791939 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.792057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.791999 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw89x\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.792614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.792524 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.793409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.792615 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.793409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.792689 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.793409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.792714 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.793409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.792755 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.793409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.792782 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.855888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.855534 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj"] Oct 31 23:11:39.860965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.860939 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:39.863657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.863628 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj"] Oct 31 23:11:39.893903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.892171 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:11:39.893903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.892259 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:11:39.893903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.892326 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:11:39.893903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.892403 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:11:39.918578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.895404 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.918931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.918897 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.919054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.919035 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.919159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.919146 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.919252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.919241 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.919360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.919347 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw89x\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.919450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.919434 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.923646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.923614 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.923850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.923833 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.923963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.923949 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.924057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.924043 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.924145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.924133 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.924258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.924245 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.924438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.924423 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.932565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.924523 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.932829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.932803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.933021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933004 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-689g8\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.933130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933117 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.933329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933314 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.933434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933420 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.933586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933570 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.933713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933699 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:39.933824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933811 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.933939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.933924 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.939715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.939687 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.939930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.939907 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjcg\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:39.940093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.940077 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.940336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.940296 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.940669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.940642 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.941677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.940960 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.942231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.942211 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.942656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.942637 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.970183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.969817 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.981639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.972904 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:39.981639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:39.977411 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041342 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041394 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041420 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041440 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041470 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041500 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.041566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041564 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041598 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041621 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041655 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041687 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041808 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-689g8\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041874 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041939 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.041967 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042006 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.042056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042043 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042069 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx4hb\" (UniqueName: \"kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042112 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042153 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042209 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjcg\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.042983 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.043891 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.044265 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.044340 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.044482 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.044754 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.045361 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.047116 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw89x\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.047411 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.047660 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.059659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.048434 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.075565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.072968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.096744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.096030 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.096744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.096674 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjcg\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.097386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.097249 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.108245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.108217 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-689g8\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.117691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.112044 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.117691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.115231 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.117691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.115691 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.150625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.149337 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.150625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.149489 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.150625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.149578 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx4hb\" (UniqueName: \"kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.150625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.150335 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.172569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.168582 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx4hb\" (UniqueName: \"kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.172980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.172953 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-062f13f9-znflj\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.199042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.198765 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:40.267881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.267846 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:40.390537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.390505 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:40.408910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.408627 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:40.428815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.426153 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:40.527656 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:11:40.527483 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:11:40.564215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.564189 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj"] Oct 31 23:11:40.593453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.593156 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd"] Oct 31 23:11:40.601809 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:40.601772 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51479f0b_8c1b_472e_b4aa_80a088062549.slice/crio-73130c78485bb8cde334b5b8c1d5c98e18bb5800a00d514890e60bfcf9d92b72 WatchSource:0}: Error finding container 73130c78485bb8cde334b5b8c1d5c98e18bb5800a00d514890e60bfcf9d92b72: Status 404 returned error can't find the container with id 73130c78485bb8cde334b5b8c1d5c98e18bb5800a00d514890e60bfcf9d92b72 Oct 31 23:11:40.604776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.604736 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.604882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.604826 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.604882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.604867 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.690565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.688272 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" event={"ID":"836a5652-b050-408b-9050-bd651a649dff","Type":"ContainerStarted","Data":"bfeadbe648dad1b4ecac242972e7086752e07322c199156f0fb0d80ecefa3e67"} Oct 31 23:11:40.691151 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.690731 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" event={"ID":"51479f0b-8c1b-472e-b4aa-80a088062549","Type":"ContainerStarted","Data":"73130c78485bb8cde334b5b8c1d5c98e18bb5800a00d514890e60bfcf9d92b72"} Oct 31 23:11:40.786595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.786565 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll"] Oct 31 23:11:40.797396 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:40.797358 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d9c0df_ff9e_4bab_8621_ee2537b266de.slice/crio-4ddc6f267c1b7be9352861d1896d2a8e16a9f1e101b7b88aec34002618a71223 WatchSource:0}: Error finding container 4ddc6f267c1b7be9352861d1896d2a8e16a9f1e101b7b88aec34002618a71223: Status 404 returned error can't find the container with id 4ddc6f267c1b7be9352861d1896d2a8e16a9f1e101b7b88aec34002618a71223 Oct 31 23:11:40.802522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.802499 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf"] Oct 31 23:11:40.807727 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:40.807687 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd24c44a_38fc_460e_a833_fd5308961380.slice/crio-b5a28576c1450b3f63f4daec8639da5cb0395d188fcbe0b3a41ff63c2509072e WatchSource:0}: Error finding container b5a28576c1450b3f63f4daec8639da5cb0395d188fcbe0b3a41ff63c2509072e: Status 404 returned error can't find the container with id b5a28576c1450b3f63f4daec8639da5cb0395d188fcbe0b3a41ff63c2509072e Oct 31 23:11:40.808435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.808377 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.808506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.808458 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.808593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.808501 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.812339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.812291 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.812600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.812572 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:40.812762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:40.812746 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:41.124501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.124454 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:41.747214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.747176 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" event={"ID":"51479f0b-8c1b-472e-b4aa-80a088062549","Type":"ContainerStarted","Data":"b338d82c0b906c30d6ac55ae97fc8626e6df193597e361a5bfe39359cf9c4f11"} Oct 31 23:11:41.759689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.758770 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" event={"ID":"cd24c44a-38fc-460e-a833-fd5308961380","Type":"ContainerStarted","Data":"c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d"} Oct 31 23:11:41.759689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.758839 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" event={"ID":"cd24c44a-38fc-460e-a833-fd5308961380","Type":"ContainerStarted","Data":"b5a28576c1450b3f63f4daec8639da5cb0395d188fcbe0b3a41ff63c2509072e"} Oct 31 23:11:41.796624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.795105 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podStartSLOduration=2.795080995 podStartE2EDuration="2.795080995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.774607678 +0000 UTC m=+1328.004162090" watchObservedRunningTime="2026-10-31 23:11:41.795080995 +0000 UTC m=+1328.024635407" Oct 31 23:11:41.799828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.799791 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" event={"ID":"e0d9c0df-ff9e-4bab-8621-ee2537b266de","Type":"ContainerStarted","Data":"e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2"} Oct 31 23:11:41.800104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.800070 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" event={"ID":"e0d9c0df-ff9e-4bab-8621-ee2537b266de","Type":"ContainerStarted","Data":"4ddc6f267c1b7be9352861d1896d2a8e16a9f1e101b7b88aec34002618a71223"} Oct 31 23:11:41.807777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.807746 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" event={"ID":"836a5652-b050-408b-9050-bd651a649dff","Type":"ContainerStarted","Data":"a827f873aaa4a2f20c2cbda89cd9e3e9dd3631275f4a11b92a8d60b2ca5c8a66"} Oct 31 23:11:41.808070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.807535 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podStartSLOduration=2.807516744 podStartE2EDuration="2.807516744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.805482687 +0000 UTC m=+1328.035037100" watchObservedRunningTime="2026-10-31 23:11:41.807516744 +0000 UTC m=+1328.037071157" Oct 31 23:11:41.911599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.910712 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podStartSLOduration=2.910680803 podStartE2EDuration="2.910680803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.858327221 +0000 UTC m=+1328.087881635" watchObservedRunningTime="2026-10-31 23:11:41.910680803 +0000 UTC m=+1328.140235215" Oct 31 23:11:41.974576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:41.973372 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" podStartSLOduration=2.973351699 podStartE2EDuration="2.973351699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:41.912985459 +0000 UTC m=+1328.142539871" watchObservedRunningTime="2026-10-31 23:11:41.973351699 +0000 UTC m=+1328.202906110" Oct 31 23:11:42.141779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.141677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:42.276804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.276757 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:42.279610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.279236 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:11:42.279610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.279307 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 23:11:42.403425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.402626 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:42.404040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.403716 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:42.404040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.403768 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:42.414690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.413021 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:42.415052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.414931 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:11:42.415052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.414984 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 23:11:42.875940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.875899 5043 generic.go:356] "Generic (PLEG): container finished" podID="836a5652-b050-408b-9050-bd651a649dff" containerID="a827f873aaa4a2f20c2cbda89cd9e3e9dd3631275f4a11b92a8d60b2ca5c8a66" exitCode=0 Oct 31 23:11:42.878060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.878029 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" event={"ID":"836a5652-b050-408b-9050-bd651a649dff","Type":"ContainerDied","Data":"a827f873aaa4a2f20c2cbda89cd9e3e9dd3631275f4a11b92a8d60b2ca5c8a66"} Oct 31 23:11:42.936405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:42.936370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:43.243656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.241602 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54"] Oct 31 23:11:43.280364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.271326 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.280364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.272413 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:11:43.280364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.272456 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 23:11:43.280364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.273015 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54"] Oct 31 23:11:43.504923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.502743 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:11:43.504923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.502806 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 23:11:43.656643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.612659 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:43.656643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.612732 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:43.656643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.652049 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.656643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.652105 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.656643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.652385 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bdp\" (UniqueName: \"kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.766642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.757788 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bdp\" (UniqueName: \"kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.766642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.757979 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.766642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.758016 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.766642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.761226 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.803122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.803088 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bdp\" (UniqueName: \"kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:43.909571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.880094 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040c1f88-c6j54\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:44.002571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:43.999370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:44.169786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.169695 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:44.356877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.322875 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:11:44.356877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.322934 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 23:11:44.408601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.403244 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:44.408601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.403311 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:44.429518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.429470 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:11:44.429664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.429573 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 23:11:44.452629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.452598 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:44.631787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.631736 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:44.632399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.632366 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:44.672005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:44.671970 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:45.038569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.037676 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:45.291085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.270047 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:11:45.375844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.342488 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:11:45.375844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.352639 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 23:11:45.404077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.404031 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:45.404269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.404090 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:45.460580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.459720 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:11:45.460580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.459792 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 23:11:45.529693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.529377 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:45.583631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.551773 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:45.651278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.591652 5043 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:45.749161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.735132 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb\" (UniqueName: \"kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb\") pod \"836a5652-b050-408b-9050-bd651a649dff\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " Oct 31 23:11:45.749161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.735310 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca\") pod \"836a5652-b050-408b-9050-bd651a649dff\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " Oct 31 23:11:45.749161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.735427 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca\") pod \"836a5652-b050-408b-9050-bd651a649dff\" (UID: \"836a5652-b050-408b-9050-bd651a649dff\") " Oct 31 23:11:45.799571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.780741 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca" pod "836a5652-b050-408b-9050-bd651a649dff" (UID: "836a5652-b050-408b-9050-bd651a649dff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:45.804447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.804408 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca" pod "836a5652-b050-408b-9050-bd651a649dff" (UID: "836a5652-b050-408b-9050-bd651a649dff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:45.808059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.808032 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb" pod "836a5652-b050-408b-9050-bd651a649dff" (UID: "836a5652-b050-408b-9050-bd651a649dff"). InnerVolumeSpecName "kube-api-access-xx4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:45.810049 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:45.810026 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb18ce55_68de_4614_8dcc_ecc80ab72dc8.slice/crio-9695fc1cec16fdc636bfe97464862c142d37187c4fe6451b91ebeffd2dfa5cd2 WatchSource:0}: Error finding container 9695fc1cec16fdc636bfe97464862c142d37187c4fe6451b91ebeffd2dfa5cd2: Status 404 returned error can't find the container with id 9695fc1cec16fdc636bfe97464862c142d37187c4fe6451b91ebeffd2dfa5cd2 Oct 31 23:11:45.907618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.907578 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 31 23:11:45.909639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.909613 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx4hb\" (UniqueName: \"kubernetes.io/projected/836a5652-b050-408b-9050-bd651a649dff-kube-api-access-xx4hb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:45.909807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.909791 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/836a5652-b050-408b-9050-bd651a649dff-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:45.909921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:45.909909 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/836a5652-b050-408b-9050-bd651a649dff-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:46.063836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.063789 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54"] Oct 31 23:11:46.264769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.264732 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" event={"ID":"836a5652-b050-408b-9050-bd651a649dff","Type":"ContainerDied","Data":"bfeadbe648dad1b4ecac242972e7086752e07322c199156f0fb0d80ecefa3e67"} Oct 31 23:11:46.265018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.264999 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfeadbe648dad1b4ecac242972e7086752e07322c199156f0fb0d80ecefa3e67" Oct 31 23:11:46.265174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.265161 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj" Oct 31 23:11:46.267817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.267789 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" event={"ID":"bb18ce55-68de-4614-8dcc-ecc80ab72dc8","Type":"ContainerStarted","Data":"9695fc1cec16fdc636bfe97464862c142d37187c4fe6451b91ebeffd2dfa5cd2"} Oct 31 23:11:46.268279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.268254 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:11:46.268435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.268411 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 23:11:46.355339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.355290 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:46.363686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.363642 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:46.364632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.364609 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:11:46.396584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.391325 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:46.396584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.391381 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:46.427573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.427457 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:11:46.430519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.430464 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:11:46.430738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.430702 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 23:11:46.902567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:46.901988 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:11:47.274570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.272649 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 23:11:47.274570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.272706 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 23:11:47.274570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.273093 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:47.392177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.392125 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:47.392473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.392421 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:47.406455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.401933 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" event={"ID":"bb18ce55-68de-4614-8dcc-ecc80ab72dc8","Type":"ContainerStarted","Data":"d15e5123c124c030b74ac62b1d773e4b45229cd4d1bfa26d3b2368c9ca14f189"} Oct 31 23:11:47.409640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.409603 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 23:11:47.409825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.409807 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 23:11:47.450486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:47.450431 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" podStartSLOduration=4.450412274 podStartE2EDuration="4.450412274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:47.447099413 +0000 UTC m=+1333.676653824" watchObservedRunningTime="2026-10-31 23:11:47.450412274 +0000 UTC m=+1333.679966689" Oct 31 23:11:48.108409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.108356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:48.391398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.391296 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 23:11:48.391398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.391363 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 23:11:48.426104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.426052 5043 generic.go:356] "Generic (PLEG): container finished" podID="bb18ce55-68de-4614-8dcc-ecc80ab72dc8" containerID="d15e5123c124c030b74ac62b1d773e4b45229cd4d1bfa26d3b2368c9ca14f189" exitCode=0 Oct 31 23:11:48.426354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.426331 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" event={"ID":"bb18ce55-68de-4614-8dcc-ecc80ab72dc8","Type":"ContainerDied","Data":"d15e5123c124c030b74ac62b1d773e4b45229cd4d1bfa26d3b2368c9ca14f189"} Oct 31 23:11:48.602621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.595858 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:48.602621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.599262 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:48.639882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.638955 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:11:48.800386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.800340 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:48.801258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.801226 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:48.802564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.802525 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:11:48.824022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:48.823254 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:49.026972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.026934 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:49.396031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.396002 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:49.396451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.396431 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:49.397223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.397197 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:11:49.519795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.519538 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:49.651292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.651221 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:49.764116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.764083 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp\" (UniqueName: \"kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp\") pod \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " Oct 31 23:11:49.764294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.764134 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca\") pod \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " Oct 31 23:11:49.764294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.764177 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca\") pod \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\" (UID: \"bb18ce55-68de-4614-8dcc-ecc80ab72dc8\") " Oct 31 23:11:49.764633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.764607 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca" pod "bb18ce55-68de-4614-8dcc-ecc80ab72dc8" (UID: "bb18ce55-68de-4614-8dcc-ecc80ab72dc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:11:49.764727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.764706 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:49.766355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.766329 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca" pod "bb18ce55-68de-4614-8dcc-ecc80ab72dc8" (UID: "bb18ce55-68de-4614-8dcc-ecc80ab72dc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:11:49.766615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.766562 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp" pod "bb18ce55-68de-4614-8dcc-ecc80ab72dc8" (UID: "bb18ce55-68de-4614-8dcc-ecc80ab72dc8"). InnerVolumeSpecName "kube-api-access-c8bdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:11:49.865578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.865512 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bdp\" (UniqueName: \"kubernetes.io/projected/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-kube-api-access-c8bdp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:49.865578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:49.865578 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb18ce55-68de-4614-8dcc-ecc80ab72dc8-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:11:50.410764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.410716 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:50.442710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.442672 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" Oct 31 23:11:50.442710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.442705 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54" event={"ID":"bb18ce55-68de-4614-8dcc-ecc80ab72dc8","Type":"ContainerDied","Data":"9695fc1cec16fdc636bfe97464862c142d37187c4fe6451b91ebeffd2dfa5cd2"} Oct 31 23:11:50.442953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.442754 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9695fc1cec16fdc636bfe97464862c142d37187c4fe6451b91ebeffd2dfa5cd2" Oct 31 23:11:50.887692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.887647 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5"] Oct 31 23:11:50.888585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.888562 5043 cpu_manager.go:414] "containerMap: removing container" podUID="836a5652-b050-408b-9050-bd651a649dff" containerName="revoke-keys" Oct 31 23:11:50.888585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.888583 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836a5652-b050-408b-9050-bd651a649dff" containerName="revoke-keys" Oct 31 23:11:50.888796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.888628 5043 cpu_manager.go:414] "containerMap: removing container" podUID="bb18ce55-68de-4614-8dcc-ecc80ab72dc8" containerName="revoke-keys" Oct 31 23:11:50.888796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.888635 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb18ce55-68de-4614-8dcc-ecc80ab72dc8" containerName="revoke-keys" Oct 31 23:11:50.888908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.888795 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb18ce55-68de-4614-8dcc-ecc80ab72dc8" containerName="revoke-keys" Oct 31 23:11:50.888908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.888817 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="836a5652-b050-408b-9050-bd651a649dff" containerName="revoke-keys" Oct 31 23:11:50.893185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.893160 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:50.899031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.899005 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-04c5c2\"/\"distinct-2-w1-04c5c2-kserve-self-signed-certs\"" Oct 31 23:11:50.906273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.906246 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5"] Oct 31 23:11:50.970985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.970925 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:50.981598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.981562 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt88z\" (UniqueName: \"kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:50.981794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.981736 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:50.981865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.981798 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:50.981979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.981950 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:50.982079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.982063 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:50.982138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:50.982125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084378 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084477 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084576 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt88z\" (UniqueName: \"kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084690 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084770 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084858 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.084938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.084912 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.085417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.085392 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.086102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.086078 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.087202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.087178 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.088108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.088085 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.098106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.098082 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt88z\" (UniqueName: \"kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z\") pod \"distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.201939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.201857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:51.213007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.212976 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:51.530776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.530746 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5"] Oct 31 23:11:51.531559 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:51.531517 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01442497_4a44_42db_bd0d_7d7839a7efe1.slice/crio-faf49198c62aa8df22d378d8a58c35582004f5ee32e9b6ab4c24b6ac0b81d41c WatchSource:0}: Error finding container faf49198c62aa8df22d378d8a58c35582004f5ee32e9b6ab4c24b6ac0b81d41c: Status 404 returned error can't find the container with id faf49198c62aa8df22d378d8a58c35582004f5ee32e9b6ab4c24b6ac0b81d41c Oct 31 23:11:51.910126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:51.910007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:52.462768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:52.462725 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" event={"ID":"01442497-4a44-42db-bd0d-7d7839a7efe1","Type":"ContainerStarted","Data":"4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd"} Oct 31 23:11:52.462922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:52.462785 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" event={"ID":"01442497-4a44-42db-bd0d-7d7839a7efe1","Type":"ContainerStarted","Data":"faf49198c62aa8df22d378d8a58c35582004f5ee32e9b6ab4c24b6ac0b81d41c"} Oct 31 23:11:52.484707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:52.484653 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podStartSLOduration=2.484639343 podStartE2EDuration="2.484639343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:52.482237639 +0000 UTC m=+1338.711792053" watchObservedRunningTime="2026-10-31 23:11:52.484639343 +0000 UTC m=+1338.714193753" Oct 31 23:11:52.607095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:52.607049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:53.213692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:53.213655 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:11:53.220314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:53.220276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:53.903803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:53.903754 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:54.256125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.256082 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs"] Oct 31 23:11:54.261664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.261630 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.271152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.271116 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs"] Oct 31 23:11:54.272251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.272206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:54.341469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341425 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.341726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341526 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.341726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341589 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.341726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341649 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.341726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341691 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvgf\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.350815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341791 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.350815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341821 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.350815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341882 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.350815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.341906 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.443120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.443078 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.443434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.443396 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvgf\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.443717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.443699 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.443848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.443833 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.444029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.444000 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.445761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.445725 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.446105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.446080 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.446232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.446175 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.446232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.446215 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.447078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.447051 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.448362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.448154 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.456638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.456606 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.457964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.457931 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.458205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.458183 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.464388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.464305 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.465598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.465380 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvgf\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.466598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.466527 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.472999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.472958 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.491029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.490745 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:54.504209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.504183 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:11:54.581476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.581395 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:54.816047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.815986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:54.933751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.933716 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs"] Oct 31 23:11:54.934246 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:11:54.934219 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6404e0_d1f4_48c9_a7be_631d72442e4e.slice/crio-a43b14a9f9e0c61c9619057bdb63afdcfff4753dd89a8cfd6448b6bb7752773e WatchSource:0}: Error finding container a43b14a9f9e0c61c9619057bdb63afdcfff4753dd89a8cfd6448b6bb7752773e: Status 404 returned error can't find the container with id a43b14a9f9e0c61c9619057bdb63afdcfff4753dd89a8cfd6448b6bb7752773e Oct 31 23:11:54.937266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.937230 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:54.937411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.937320 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:54.937411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:54.937361 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:11:55.505084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:55.504468 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" event={"ID":"aa6404e0-d1f4-48c9-a7be-631d72442e4e","Type":"ContainerStarted","Data":"cb29e637a0267be632d53370dd52f10f13c4ef66db6297b1ce425f61d4c631d6"} Oct 31 23:11:55.505084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:55.504540 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" event={"ID":"aa6404e0-d1f4-48c9-a7be-631d72442e4e","Type":"ContainerStarted","Data":"a43b14a9f9e0c61c9619057bdb63afdcfff4753dd89a8cfd6448b6bb7752773e"} Oct 31 23:11:55.533266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:55.533200 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podStartSLOduration=1.533178297 podStartE2EDuration="1.533178297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:11:55.530105062 +0000 UTC m=+1341.759659473" watchObservedRunningTime="2026-10-31 23:11:55.533178297 +0000 UTC m=+1341.762732707" Oct 31 23:11:55.582161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:55.582118 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:11:55.594667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:55.594620 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:56.309817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:56.309780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:56.584305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:56.584216 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:11:56.584305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:56.584270 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:11:57.092575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:57.092187 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:57.582655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:57.582612 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:11:57.583062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:57.582682 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:11:57.710244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:57.710202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:58.560119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:58.560075 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:58.582723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:58.582667 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:11:58.583329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:58.583291 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:11:59.543574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:59.536024 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:11:59.639896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:59.639361 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:11:59.639896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:59.639445 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:12:00.005630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:11:59.979927 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.028979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.028664 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.364890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.363640 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.455512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.455464 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.498904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.498850 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.500255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.500220 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.589033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.588988 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:12:00.589312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.589279 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:12:00.612840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.612785 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:00.616903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.616809 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:12:00.906838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:00.906724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:12:01.028570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.028211 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:01.036571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.032631 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:01.036571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.033100 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:01.273571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.265629 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:01.373645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.371866 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:01.373645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.372683 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:01.391594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.391499 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:01.494570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.461751 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:01.535329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.514060 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:01.535329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.514108 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:01.535329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.525215 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:01.535329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.526607 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:01.535329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.527644 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:01.612834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.612685 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:12:01.612834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.612760 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:12:01.644607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.626319 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:12:01.644607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.627462 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:01.922793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:01.922662 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:02.089569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.085638 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:02.223631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.200789 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:02.278109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.278060 5043 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 23:12:02.551573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.547886 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:02.616760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.574804 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:12:02.617835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.584076 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:12:02.618058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.618025 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:12:02.651040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.637679 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:12:02.651040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.645841 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:02.676317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.676271 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:02.698289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.697704 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:12:02.698652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.698189 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 23:12:02.729607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.728943 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:02.743750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.743709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:02.999734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:02.999679 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:03.104812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.104756 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:03.135362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.127651 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:03.397632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.333621 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:03.437703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.437663 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:03.582411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.581907 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:12:03.582411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.581989 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:12:03.626589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.624797 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:03.626589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.625150 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:03.645941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.644908 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:03.645941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.645614 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:12:03.646594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.646541 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:03.705625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.699660 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:12:03.992346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.992318 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:12:03.992607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:03.992578 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:04.052614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.048480 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:04.052614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.048567 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:04.192834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.192787 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:04.209782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.204271 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:04.283254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.283146 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:04.299785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.284407 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:04.300191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.300166 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:04.440524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.440368 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:04.443256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.443232 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:04.448108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.448089 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:12:04.582597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.582489 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 23:12:04.582597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.582568 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 23:12:04.923602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:04.921393 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:05.537575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:05.537468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:05.647080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:05.647036 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:12:05.648390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:05.648369 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:12:05.649112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:05.649092 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:12:06.075171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.075135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:06.318570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.317334 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:06.673255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.673220 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w"] Oct 31 23:12:06.680690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.679283 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.685124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.685091 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:12:06.686292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.686268 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:12:06.686702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.686666 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w"] Oct 31 23:12:06.691383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.690956 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:12:06.691383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.691035 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:12:06.791507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.791464 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.791712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.791664 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.791813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.791789 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgj7\" (UniqueName: \"kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.893073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.893035 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.893261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.893107 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgj7\" (UniqueName: \"kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.893261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.893224 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.894111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.894084 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.895475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.895453 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:06.906733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:06.906706 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgj7\" (UniqueName: \"kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7\") pod \"maas-api-revoke-sub-f880724d-4pc4w\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:07.007607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.007571 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:07.100891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.099726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:07.343712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.343677 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w"] Oct 31 23:12:07.795489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.795447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:07.954865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.954780 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" event={"ID":"796a157b-aa08-4e52-bf87-5d6325dd4af1","Type":"ContainerStarted","Data":"3ce0927ddc223e937937853d98509e409aa10f726fa807647825ead8144ca080"} Oct 31 23:12:07.954865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.954834 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" event={"ID":"796a157b-aa08-4e52-bf87-5d6325dd4af1","Type":"ContainerStarted","Data":"7826c9164c89a1bacc64736b5a60087aff96ffe1330ad114457f07f3936de1dd"} Oct 31 23:12:07.977837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:07.977790 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" podStartSLOduration=1.977777927 podStartE2EDuration="1.977777927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:07.973992317 +0000 UTC m=+1354.203546745" watchObservedRunningTime="2026-10-31 23:12:07.977777927 +0000 UTC m=+1354.207332338" Oct 31 23:12:08.462252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:08.462212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:08.963787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:08.963754 5043 generic.go:356] "Generic (PLEG): container finished" podID="796a157b-aa08-4e52-bf87-5d6325dd4af1" containerID="3ce0927ddc223e937937853d98509e409aa10f726fa807647825ead8144ca080" exitCode=0 Oct 31 23:12:08.964163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:08.963832 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" event={"ID":"796a157b-aa08-4e52-bf87-5d6325dd4af1","Type":"ContainerDied","Data":"3ce0927ddc223e937937853d98509e409aa10f726fa807647825ead8144ca080"} Oct 31 23:12:09.107748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.107703 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:09.275949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.275917 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:09.470323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.470290 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq"] Oct 31 23:12:09.475362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.475336 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.496467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.496435 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq"] Oct 31 23:12:09.560434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560357 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560402 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560421 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560584 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llpk\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560673 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560695 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560795 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560914 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560901 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.560968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.560955 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.662814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.662771 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.662978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.662928 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.662978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.662962 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.662984 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663018 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llpk\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663052 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663078 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663134 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663246 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663358 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663428 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663476 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663509 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.663731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.663703 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.665514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.665487 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.665807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.665787 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.675748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.675723 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.675842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.675761 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llpk\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.799274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.799244 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:09.938934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:09.936974 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:10.116604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.116578 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq"] Oct 31 23:12:10.133690 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:10.133656 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cbfec0_981c_4be5_b984_0f1b8d71cc59.slice/crio-944d3bdda09e6dfc66cdd2cbea209d11c2b233d9b5c6960a14e95370855da153 WatchSource:0}: Error finding container 944d3bdda09e6dfc66cdd2cbea209d11c2b233d9b5c6960a14e95370855da153: Status 404 returned error can't find the container with id 944d3bdda09e6dfc66cdd2cbea209d11c2b233d9b5c6960a14e95370855da153 Oct 31 23:12:10.137784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.137740 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:10.137894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.137838 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:10.137894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.137884 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:10.184227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.184206 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:10.290112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.290084 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca\") pod \"796a157b-aa08-4e52-bf87-5d6325dd4af1\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " Oct 31 23:12:10.290306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.290162 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca\") pod \"796a157b-aa08-4e52-bf87-5d6325dd4af1\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " Oct 31 23:12:10.290306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.290203 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7\" (UniqueName: \"kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7\") pod \"796a157b-aa08-4e52-bf87-5d6325dd4af1\" (UID: \"796a157b-aa08-4e52-bf87-5d6325dd4af1\") " Oct 31 23:12:10.290829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.290797 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca" pod "796a157b-aa08-4e52-bf87-5d6325dd4af1" (UID: "796a157b-aa08-4e52-bf87-5d6325dd4af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:12:10.293170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.293141 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7" pod "796a157b-aa08-4e52-bf87-5d6325dd4af1" (UID: "796a157b-aa08-4e52-bf87-5d6325dd4af1"). InnerVolumeSpecName "kube-api-access-gqgj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:12:10.293819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.293429 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca" pod "796a157b-aa08-4e52-bf87-5d6325dd4af1" (UID: "796a157b-aa08-4e52-bf87-5d6325dd4af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:12:10.392304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.392221 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:10.392304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.392251 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796a157b-aa08-4e52-bf87-5d6325dd4af1-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:10.392304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.392260 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgj7\" (UniqueName: \"kubernetes.io/projected/796a157b-aa08-4e52-bf87-5d6325dd4af1-kube-api-access-gqgj7\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:10.457531 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:12:10.457488 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:12:10.823754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:10.822809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:11.003639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.003595 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" event={"ID":"41cbfec0-981c-4be5-b984-0f1b8d71cc59","Type":"ContainerStarted","Data":"35469d41d88c962f7545fe8df512a334b4e1c24bd26d8c77412b355fac00cf50"} Oct 31 23:12:11.003848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.003664 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" event={"ID":"41cbfec0-981c-4be5-b984-0f1b8d71cc59","Type":"ContainerStarted","Data":"944d3bdda09e6dfc66cdd2cbea209d11c2b233d9b5c6960a14e95370855da153"} Oct 31 23:12:11.005234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.005202 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" event={"ID":"796a157b-aa08-4e52-bf87-5d6325dd4af1","Type":"ContainerDied","Data":"7826c9164c89a1bacc64736b5a60087aff96ffe1330ad114457f07f3936de1dd"} Oct 31 23:12:11.005368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.005246 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7826c9164c89a1bacc64736b5a60087aff96ffe1330ad114457f07f3936de1dd" Oct 31 23:12:11.005368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.005211 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w" Oct 31 23:12:11.050661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.050539 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podStartSLOduration=2.050521029 podStartE2EDuration="2.050521029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:11.038414573 +0000 UTC m=+1357.267968994" watchObservedRunningTime="2026-10-31 23:12:11.050521029 +0000 UTC m=+1357.280075443" Oct 31 23:12:11.244640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.241493 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:11.257236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.257157 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:11.685429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.685328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:11.800202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.800170 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:11.801483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.801457 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 23:12:11.801619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:11.801509 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 23:12:12.447141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.447101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:12.770581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.763675 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6"] Oct 31 23:12:12.770581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.764254 5043 cpu_manager.go:414] "containerMap: removing container" podUID="796a157b-aa08-4e52-bf87-5d6325dd4af1" containerName="revoke-keys" Oct 31 23:12:12.770581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.764266 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796a157b-aa08-4e52-bf87-5d6325dd4af1" containerName="revoke-keys" Oct 31 23:12:12.770581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.764403 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="796a157b-aa08-4e52-bf87-5d6325dd4af1" containerName="revoke-keys" Oct 31 23:12:12.770581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.769490 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.774525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.774498 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"distinct-w0-a16dfb-kserve-self-signed-certs\"" Oct 31 23:12:12.776685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.776645 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6"] Oct 31 23:12:12.805136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.805083 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 23:12:12.805302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.805225 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 23:12:12.851295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.851248 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:12.877595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.877558 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.877744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.877643 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.877744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.877671 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.877884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.877853 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdg6l\" (UniqueName: \"kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.877969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.877948 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.878013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.877986 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.989190 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.989255 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.989281 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.989406 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdg6l\" (UniqueName: \"kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.989462 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.989495 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.990139 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.990402 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.992538 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:12.995570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:12.993247 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:13.037614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.034677 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:13.044566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.042408 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdg6l\" (UniqueName: \"kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l\") pod \"distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:13.091705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.091670 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:13.412374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.412287 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:13.483884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.481381 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6"] Oct 31 23:12:13.487454 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:13.487124 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod013ef212_2c66_476e_91fb_38ac1d259d38.slice/crio-5dbb668beac83d30ec01ad2ec22f97cd6db65e2c39ea09b3794d58ac73982254 WatchSource:0}: Error finding container 5dbb668beac83d30ec01ad2ec22f97cd6db65e2c39ea09b3794d58ac73982254: Status 404 returned error can't find the container with id 5dbb668beac83d30ec01ad2ec22f97cd6db65e2c39ea09b3794d58ac73982254 Oct 31 23:12:13.800599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.800564 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 23:12:13.800775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:13.800621 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 23:12:14.059313 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:14.059232 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" event={"ID":"013ef212-2c66-476e-91fb-38ac1d259d38","Type":"ContainerStarted","Data":"76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963"} Oct 31 23:12:14.059485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:14.059471 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" event={"ID":"013ef212-2c66-476e-91fb-38ac1d259d38","Type":"ContainerStarted","Data":"5dbb668beac83d30ec01ad2ec22f97cd6db65e2c39ea09b3794d58ac73982254"} Oct 31 23:12:14.093023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:14.092968 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:14.352644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:14.352336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:14.800472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:14.800438 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 23:12:14.800896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:14.800490 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 23:12:15.277824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:15.277782 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:15.799874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:15.799839 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 23:12:15.800066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:15.799914 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 23:12:16.150190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:16.150095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:16.788342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:16.788286 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:16.800245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:16.800215 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 23:12:16.800424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:16.800266 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 23:12:16.988700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:16.988634 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:17.142101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.141968 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podStartSLOduration=5.141946075 podStartE2EDuration="5.141946075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:14.081615287 +0000 UTC m=+1360.311169699" watchObservedRunningTime="2026-10-31 23:12:17.141946075 +0000 UTC m=+1363.371500486" Oct 31 23:12:17.142519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.142497 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt"] Oct 31 23:12:17.156572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.156526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:17.160807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.160776 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.168649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.168418 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2294a7\"/\"distinct-2-w5-2294a7-kserve-self-signed-certs\"" Oct 31 23:12:17.173382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.173339 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt"] Oct 31 23:12:17.290255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.290219 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc856\" (UniqueName: \"kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.290255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.290263 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.290528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.290310 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.290610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.290569 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.290733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.290692 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.290863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.290745 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.300146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.300101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:17.391561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.391497 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.391805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.391726 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.391956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.391935 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.392227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.392163 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc856\" (UniqueName: \"kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.392381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.392365 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.392968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.392637 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.393104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.392828 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.393227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.392203 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.393369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.393350 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.400831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.398572 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.420080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.419961 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.438568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.422489 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc856\" (UniqueName: \"kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856\") pod \"distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.491570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.484822 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:17.722715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.720421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:17.804181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.804144 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:17.805267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.805241 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:17.806109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.806088 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:12:17.810126 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:17.810102 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9bbad83_e55f_4905_941c_cc28506303ed.slice/crio-0eb7bab1c542d7de5e1196b893191f268b577597b154f6bbacc852b0ce8f6526 WatchSource:0}: Error finding container 0eb7bab1c542d7de5e1196b893191f268b577597b154f6bbacc852b0ce8f6526: Status 404 returned error can't find the container with id 0eb7bab1c542d7de5e1196b893191f268b577597b154f6bbacc852b0ce8f6526 Oct 31 23:12:17.811152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.811130 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt"] Oct 31 23:12:17.838323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.838298 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w"] Oct 31 23:12:17.844326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.844288 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:17.848901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.848879 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w"] Oct 31 23:12:17.850755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.850729 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:12:17.850938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.850923 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:12:17.851136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.851121 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:12:17.959467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:17.952395 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:12:18.009900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.009233 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.009900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.009429 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmr7b\" (UniqueName: \"kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.009900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.009616 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.107733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.107685 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" event={"ID":"d9bbad83-e55f-4905-941c-cc28506303ed","Type":"ContainerStarted","Data":"897efee68277d18102448bf405190583a7482170251dcd805e74f056d02d63a0"} Oct 31 23:12:18.107925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.107755 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" event={"ID":"d9bbad83-e55f-4905-941c-cc28506303ed","Type":"ContainerStarted","Data":"0eb7bab1c542d7de5e1196b893191f268b577597b154f6bbacc852b0ce8f6526"} Oct 31 23:12:18.110975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.110945 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.111193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.111171 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.111426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.111397 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmr7b\" (UniqueName: \"kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.111768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.111743 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.113940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.113913 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.123600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.123578 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmr7b\" (UniqueName: \"kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b\") pod \"maas-api-revoke-sub-5b3cc357-wfr7w\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.131074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.131030 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podStartSLOduration=1.13101812 podStartE2EDuration="1.13101812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:18.127644298 +0000 UTC m=+1364.357198708" watchObservedRunningTime="2026-10-31 23:12:18.13101812 +0000 UTC m=+1364.360572531" Oct 31 23:12:18.185403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.185369 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:18.483253 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:18.483218 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b2e27f_2e7b_4e71_be22_b8cf3c7df578.slice/crio-3f661d0bf5ab168c6aa112ddff17d20745f3eed3d6d1bf288a413f564c8f926c WatchSource:0}: Error finding container 3f661d0bf5ab168c6aa112ddff17d20745f3eed3d6d1bf288a413f564c8f926c: Status 404 returned error can't find the container with id 3f661d0bf5ab168c6aa112ddff17d20745f3eed3d6d1bf288a413f564c8f926c Oct 31 23:12:18.483889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.483864 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w"] Oct 31 23:12:18.485229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.485196 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:18.503962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:18.503921 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:19.115948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:19.115824 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" event={"ID":"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578","Type":"ContainerStarted","Data":"f9badd16f34b7a57c1a5697b80f2f3279d3521048a22efa7436090d022cfbafd"} Oct 31 23:12:19.115948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:19.115892 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" event={"ID":"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578","Type":"ContainerStarted","Data":"3f661d0bf5ab168c6aa112ddff17d20745f3eed3d6d1bf288a413f564c8f926c"} Oct 31 23:12:19.146561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:19.146503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:19.161234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:19.161170 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" podStartSLOduration=2.161149553 podStartE2EDuration="2.161149553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:19.146273862 +0000 UTC m=+1365.375828275" watchObservedRunningTime="2026-10-31 23:12:19.161149553 +0000 UTC m=+1365.390703966" Oct 31 23:12:19.725742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:19.725655 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:20.127533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:20.127493 5043 generic.go:356] "Generic (PLEG): container finished" podID="f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" containerID="f9badd16f34b7a57c1a5697b80f2f3279d3521048a22efa7436090d022cfbafd" exitCode=0 Oct 31 23:12:20.127820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:20.127580 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" event={"ID":"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578","Type":"ContainerDied","Data":"f9badd16f34b7a57c1a5697b80f2f3279d3521048a22efa7436090d022cfbafd"} Oct 31 23:12:20.420141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:20.420038 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:21.129569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.117337 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:21.353563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.353529 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:21.472596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.472484 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca\") pod \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " Oct 31 23:12:21.472596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.472577 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca\") pod \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " Oct 31 23:12:21.472782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.472615 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b\" (UniqueName: \"kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b\") pod \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\" (UID: \"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578\") " Oct 31 23:12:21.472912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.472891 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca" pod "f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" (UID: "f8b2e27f-2e7b-4e71-be22-b8cf3c7df578"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:12:21.473305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.473287 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:21.474899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.474875 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca" pod "f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" (UID: "f8b2e27f-2e7b-4e71-be22-b8cf3c7df578"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:12:21.474974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.474913 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b" pod "f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" (UID: "f8b2e27f-2e7b-4e71-be22-b8cf3c7df578"). InnerVolumeSpecName "kube-api-access-wmr7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:12:21.575301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.575261 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:21.575301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.575294 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmr7b\" (UniqueName: \"kubernetes.io/projected/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578-kube-api-access-wmr7b\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:21.903336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:21.903267 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:22.166197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:22.166117 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" Oct 31 23:12:22.166197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:22.166155 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w" event={"ID":"f8b2e27f-2e7b-4e71-be22-b8cf3c7df578","Type":"ContainerDied","Data":"3f661d0bf5ab168c6aa112ddff17d20745f3eed3d6d1bf288a413f564c8f926c"} Oct 31 23:12:22.166624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:22.166207 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f661d0bf5ab168c6aa112ddff17d20745f3eed3d6d1bf288a413f564c8f926c" Oct 31 23:12:22.622906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:22.622214 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:23.093624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:23.093584 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:23.107330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:23.107302 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:23.184233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:23.184210 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:23.212817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:23.212784 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:23.911472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:23.911424 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:24.251020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.250982 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:24.604940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.604854 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6"] Oct 31 23:12:24.606085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.606059 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" containerName="revoke-keys" Oct 31 23:12:24.606241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.606228 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" containerName="revoke-keys" Oct 31 23:12:24.606579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.606564 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" containerName="revoke-keys" Oct 31 23:12:24.638568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.633512 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6"] Oct 31 23:12:24.638568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.633684 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.723905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.723872 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzv8\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.724234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.724213 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.724385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.724371 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.724513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.724501 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.724746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.724732 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.724884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.724872 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.725574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.725045 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.738722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.738674 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.738918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.738856 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.788520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.788468 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb"] Oct 31 23:12:24.795539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.794728 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.810743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.810692 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb"] Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.853465 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.853616 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.853653 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.853743 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.853777 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.853896 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.854007 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzv8\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.854061 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.855624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.854091 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.867015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.856602 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.904311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.904239 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.905067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.905019 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.905307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.905218 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.961687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.906838 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.963878 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.963965 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.963998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.964067 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.964146 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.964205 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.964253 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2txn\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.964450 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.965069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.964494 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:24.986568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.975132 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:24.986568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.975752 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:24.986568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:24.975983 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:25.041204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.011181 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072093 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072161 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072248 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072338 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072402 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072434 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2txn\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072627 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072676 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.072723 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.073231 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.089039 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.089433 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.089816 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.093579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.089877 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs"] Oct 31 23:12:25.120639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.105262 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs"] Oct 31 23:12:25.120639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.105421 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.120639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.107346 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247379 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247490 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247579 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247686 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247733 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247762 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gl9\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247838 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247879 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.249573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.247971 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.259574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.256416 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.270688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.263876 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.270688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.266622 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzv8\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:25.316496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.316200 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:25.378518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.378412 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2txn\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.417569 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.417636 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.417671 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gl9\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.417753 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.417792 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.417873 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.418026 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.418104 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.418171 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.418856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.418831 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.419139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.419113 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.419405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.419381 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.427693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.424584 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.428604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.428509 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.480570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.478206 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.480570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.478532 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gl9\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.488574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.485269 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.488574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.485795 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token\") pod \"e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:25.504569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.503470 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token\") pod \"e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.721933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.721830 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:25.809568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:25.807378 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:26.193571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.187889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:26.254673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.245637 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:12:26.563745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.563652 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:12:26.629210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.629172 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:26.678725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.678664 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:26.680769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.680731 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:26.702887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.702849 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:26.723569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.721738 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6"] Oct 31 23:12:26.819684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.819593 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:26.819931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.819908 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:26.820085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:26.820067 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:27.021939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.018886 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.179632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.177074 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.237571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.234674 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.270809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.269604 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:27.273271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.273236 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:12:27.290572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.275245 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:27.462041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.383243 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:12:27.496576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.489215 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.497270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.488434 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:27.552608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.440521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:27.602938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.602647 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.657086 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.698756 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.698794 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.754463 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.771622 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" event={"ID":"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba","Type":"ContainerStarted","Data":"dc9c4d3d15ec3238b22dac69d0450d456b4adab4a550e6c37de2890ad68195dc"} Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.776960 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.777010 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.777044 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.777074 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:27.785440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:27.777106 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:28.113484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.111121 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:28.125589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.114007 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:28.161578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.159812 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:28.183396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.166628 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:28.200221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.200187 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:28.201629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.200785 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:28.593113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.568633 5043 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:12:28.601675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.593756 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:12:28.614453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.614325 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:28.799206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.799160 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:12:28.802129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.802047 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:28.806874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.806839 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:28.809720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.809681 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:28.809830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.809800 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:28.809898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.809857 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.819579 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.819638 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.819674 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.819718 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.819736 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs"] Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.825896 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.825946 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:28.839737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.828628 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:28.903119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.903089 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:28.920005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.919974 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:28.931844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.931724 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:28.956692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.956643 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:12:28.957245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.957225 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl"] Oct 31 23:12:28.982920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.982889 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl"] Oct 31 23:12:28.983241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.983223 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb"] Oct 31 23:12:28.983414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.983162 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:28.985127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.985095 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:28.987032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.987002 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:28.987185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.987166 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:28.987774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.987757 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:12:28.988012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.987997 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:12:28.989766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:28.989738 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:12:29.005886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.000245 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" event={"ID":"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba","Type":"ContainerStarted","Data":"0031786746efc4e173db89f8d083137c09cae82b1884156665e1f8e327d516ab"} Oct 31 23:12:29.007909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.007882 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" event={"ID":"d336aa46-fc0a-4c34-a9dc-a45949939314","Type":"ContainerStarted","Data":"8b141b09db3b33868298427eacbfeefe7c4824043ad86b682ce0e4ac6cc3ed74"} Oct 31 23:12:29.010393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.010368 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" event={"ID":"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f","Type":"ContainerStarted","Data":"4f8d7e4668aa540ed494ae32b4d84f23d4cd7a55993fed7ebf05e7b6eec2d93a"} Oct 31 23:12:29.088019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.088000 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:12:29.091492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.090820 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fbm\" (UniqueName: \"kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.091492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.090889 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.091492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.091056 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.192591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.192465 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fbm\" (UniqueName: \"kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.192727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.192637 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.193018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.192997 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.198228 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.195486 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.223422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.217235 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.223422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.218429 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fbm\" (UniqueName: \"kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm\") pod \"maas-api-revoke-sub-4c7a3f8e-cxprl\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.272241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.271511 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podStartSLOduration=5.271478581 podStartE2EDuration="5.271478581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:29.264191729 +0000 UTC m=+1375.493746147" watchObservedRunningTime="2026-10-31 23:12:29.271478581 +0000 UTC m=+1375.501032993" Oct 31 23:12:29.358480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.338676 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:29.358480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.338962 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:29.380434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.379866 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:29.380434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.379927 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:29.715390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:29.715337 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl"] Oct 31 23:12:29.749700 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:29.749654 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58ee5e4_a240_490e_96ee_5019a606c74e.slice/crio-5ba533bc6b8383490de1bdf0ea824ff0e14d5b43e54f69ca66880065caea7911 WatchSource:0}: Error finding container 5ba533bc6b8383490de1bdf0ea824ff0e14d5b43e54f69ca66880065caea7911: Status 404 returned error can't find the container with id 5ba533bc6b8383490de1bdf0ea824ff0e14d5b43e54f69ca66880065caea7911 Oct 31 23:12:30.060993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.059253 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" event={"ID":"d336aa46-fc0a-4c34-a9dc-a45949939314","Type":"ContainerStarted","Data":"0570764552fa2ecedac368b40747d85ec5da96307a5f7a59731701891cd2befa"} Oct 31 23:12:30.062597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.062569 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" event={"ID":"c58ee5e4-a240-490e-96ee-5019a606c74e","Type":"ContainerStarted","Data":"5ba533bc6b8383490de1bdf0ea824ff0e14d5b43e54f69ca66880065caea7911"} Oct 31 23:12:30.065724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.065704 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" event={"ID":"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f","Type":"ContainerStarted","Data":"60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50"} Oct 31 23:12:30.262225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.261821 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podStartSLOduration=6.261798488 podStartE2EDuration="6.261798488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:30.199658819 +0000 UTC m=+1376.429213231" watchObservedRunningTime="2026-10-31 23:12:30.261798488 +0000 UTC m=+1376.491352901" Oct 31 23:12:30.321569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.320889 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.321569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.320949 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:30.724629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.724521 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:30.725307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.725283 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.725463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.725441 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:30.810410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.809509 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:30.810616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.810474 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:30.810616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:30.810526 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:31.132961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.132091 5043 generic.go:356] "Generic (PLEG): container finished" podID="c58ee5e4-a240-490e-96ee-5019a606c74e" containerID="6050e19026abbfd05d532c1d7a11bb04d493d695f770512ded4fe0de3a0298d0" exitCode=0 Oct 31 23:12:31.139575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.133736 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" event={"ID":"c58ee5e4-a240-490e-96ee-5019a606c74e","Type":"ContainerDied","Data":"6050e19026abbfd05d532c1d7a11bb04d493d695f770512ded4fe0de3a0298d0"} Oct 31 23:12:31.237627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.236971 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podStartSLOduration=7.236949472 podStartE2EDuration="7.236949472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:30.269075769 +0000 UTC m=+1376.498630183" watchObservedRunningTime="2026-10-31 23:12:31.236949472 +0000 UTC m=+1377.466503883" Oct 31 23:12:31.341570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.340650 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.341570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.340710 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:31.511412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.507498 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:31.726569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.725652 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.726569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.725728 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:31.812628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.809294 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:31.812628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.809351 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:31.840946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:31.835028 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:32.162902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.152397 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:32.162902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.162096 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.177572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.163277 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.269583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.238685 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:32.362345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.362311 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.362881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.362678 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:32.723336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.722906 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.723336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.722968 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:32.809900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.809853 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:32.810453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:32.810407 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:33.339003 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.337046 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.339003 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.337106 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:33.480963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.480862 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" event={"ID":"c58ee5e4-a240-490e-96ee-5019a606c74e","Type":"ContainerDied","Data":"5ba533bc6b8383490de1bdf0ea824ff0e14d5b43e54f69ca66880065caea7911"} Oct 31 23:12:33.480963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.480928 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ba533bc6b8383490de1bdf0ea824ff0e14d5b43e54f69ca66880065caea7911" Oct 31 23:12:33.526324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.526289 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:33.685634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.673108 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca\") pod \"c58ee5e4-a240-490e-96ee-5019a606c74e\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " Oct 31 23:12:33.685634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.673341 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm\" (UniqueName: \"kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm\") pod \"c58ee5e4-a240-490e-96ee-5019a606c74e\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " Oct 31 23:12:33.685634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.673390 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca\") pod \"c58ee5e4-a240-490e-96ee-5019a606c74e\" (UID: \"c58ee5e4-a240-490e-96ee-5019a606c74e\") " Oct 31 23:12:33.687303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.686993 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca" pod "c58ee5e4-a240-490e-96ee-5019a606c74e" (UID: "c58ee5e4-a240-490e-96ee-5019a606c74e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:12:33.732763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.731067 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.732763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.731120 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:33.840602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.839670 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:33.840602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.839727 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:33.840602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.840478 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:33.841207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.840619 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:33.890036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.888923 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca" pod "c58ee5e4-a240-490e-96ee-5019a606c74e" (UID: "c58ee5e4-a240-490e-96ee-5019a606c74e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:12:33.903205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.903138 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:33.903205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.903175 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58ee5e4-a240-490e-96ee-5019a606c74e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:34.149570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:33.971618 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.149570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.093630 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:12:34.149570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.098622 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.270661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.216465 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.300079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.275189 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.339688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.329380 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm" pod "c58ee5e4-a240-490e-96ee-5019a606c74e" (UID: "c58ee5e4-a240-490e-96ee-5019a606c74e"). InnerVolumeSpecName "kube-api-access-p7fbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:12:34.475624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.367744 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.475624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.385643 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:34.475624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.347252 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:34.475624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.401988 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7fbm\" (UniqueName: \"kubernetes.io/projected/c58ee5e4-a240-490e-96ee-5019a606c74e-kube-api-access-p7fbm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:34.727626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.703428 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:12:34.730121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.729717 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.730354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.730315 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:34.810578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.810526 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:34.810937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.810894 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:34.841140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.841100 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.842953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.842928 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:34.851594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.842968 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:12:34.852499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.852473 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:34.874618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:34.870520 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl" Oct 31 23:12:35.018573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.016621 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.135596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.108039 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.135596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.108578 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:12:35.135596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.113277 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:35.135596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.113347 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:35.135596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.113388 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:35.223724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.217660 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:12:35.223724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.218281 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:35.272626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.239350 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:35.275484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.275422 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:12:35.276159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.276091 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:35.335862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.335828 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.336172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.336129 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:35.337733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.337075 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:35.407017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.406967 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.407660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.407640 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:35.742199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.740033 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.742199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.740088 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:35.775608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.773912 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:35.812920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.812738 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:35.812920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.812896 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:35.853715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.853639 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:35.883579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.883531 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:35.937620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:35.937577 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.216567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.197045 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.226573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.226224 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:36.261723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.261678 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.281160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.281121 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:12:36.282042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.280247 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:12:36.283356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.280532 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:12:36.340713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.340665 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:36.341405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.341376 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.341616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.341592 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:36.390655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.390611 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:36.390862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.390753 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:36.429116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.429071 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:36.458708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.458666 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:36.542609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.535718 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:36.589330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.589290 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:36.589527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.589373 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:36.730691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.729991 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.730691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.730054 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:36.844127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.842374 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 23:12:36.844127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:36.842436 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 23:12:37.321076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.321035 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.322576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.322280 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:12:37.502491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.500997 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:37.546703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.546665 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:37.731664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.731424 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 23:12:37.731664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.731523 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 23:12:37.979570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:37.973103 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:38.297569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.295986 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:38.297778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.297608 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:38.299627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.299604 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:12:38.578618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.571978 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:38.578618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.573721 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:38.588567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.585827 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:12:38.881083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.880993 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:38.890947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.890747 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:38.909500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:38.909467 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:12:39.363751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.363715 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:39.731338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.731265 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg"] Oct 31 23:12:39.732204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.732186 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c58ee5e4-a240-490e-96ee-5019a606c74e" containerName="revoke-keys" Oct 31 23:12:39.732243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.732207 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58ee5e4-a240-490e-96ee-5019a606c74e" containerName="revoke-keys" Oct 31 23:12:39.732421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.732410 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58ee5e4-a240-490e-96ee-5019a606c74e" containerName="revoke-keys" Oct 31 23:12:39.751659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.751629 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.758530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.758493 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg"] Oct 31 23:12:39.803695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.803476 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.803849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.803752 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.803849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.803840 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.803982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.803955 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.804036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.804007 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.804121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.804102 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtfrj\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.804171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.804156 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.804227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.804212 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.804276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.804240 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.905969 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906007 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906089 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906123 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906298 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906379 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906456 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906429 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906486 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906490 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906499 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtfrj\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906655 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906711 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.906860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.906835 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.907185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.907161 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.908883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.908859 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.909227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.909210 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.918079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.918053 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:39.918295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:39.918278 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtfrj\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:40.072059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:40.072020 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:40.374037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:40.374007 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg"] Oct 31 23:12:40.374178 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:40.374139 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b6666c_4eae_49da_9e9b_d2806ae3ed50.slice/crio-ef85cf096a215b263ff479c987c37626d3f9f2fb09c84b8a31886b1609c1913a WatchSource:0}: Error finding container ef85cf096a215b263ff479c987c37626d3f9f2fb09c84b8a31886b1609c1913a: Status 404 returned error can't find the container with id ef85cf096a215b263ff479c987c37626d3f9f2fb09c84b8a31886b1609c1913a Oct 31 23:12:40.377104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:40.377071 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:40.377179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:40.377141 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:40.377179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:40.377171 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:40.460910 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:12:40.460139 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:12:40.639250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:40.637120 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:41.255732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:41.255698 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" event={"ID":"d0b6666c-4eae-49da-9e9b-d2806ae3ed50","Type":"ContainerStarted","Data":"e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0"} Oct 31 23:12:41.256212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:41.255750 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" event={"ID":"d0b6666c-4eae-49da-9e9b-d2806ae3ed50","Type":"ContainerStarted","Data":"ef85cf096a215b263ff479c987c37626d3f9f2fb09c84b8a31886b1609c1913a"} Oct 31 23:12:41.282691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:41.282647 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podStartSLOduration=2.282634338 podStartE2EDuration="2.282634338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:41.279226206 +0000 UTC m=+1387.508780620" watchObservedRunningTime="2026-10-31 23:12:41.282634338 +0000 UTC m=+1387.512188748" Oct 31 23:12:42.072305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:42.072263 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:42.073471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:42.073446 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:12:42.073625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:42.073501 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 23:12:42.133571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:42.133051 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:42.842568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:42.842248 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:43.072735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:43.072366 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:12:43.072735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:43.072417 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 23:12:43.528140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:43.527989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:44.073259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:44.073220 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:12:44.073691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:44.073271 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 23:12:44.291842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:44.291801 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:44.936570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:44.934595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:45.073088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:45.073054 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:12:45.073297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:45.073110 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 23:12:45.604536 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:45.604508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:46.073344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.073307 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:12:46.073767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.073364 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 23:12:46.317570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.314222 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:46.522397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.522357 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m"] Oct 31 23:12:46.546567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.532644 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.554612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.554569 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a16dfb\"/\"distinct-2-w0-a16dfb-kserve-self-signed-certs\"" Oct 31 23:12:46.561733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.561699 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m"] Oct 31 23:12:46.595421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.595367 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:46.676792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.675350 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.676792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.675575 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.676792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.675736 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.676792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.675770 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.676792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.675888 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p897\" (UniqueName: \"kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.676792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.676070 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.778368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.778273 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.778368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.778327 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.778638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.778390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p897\" (UniqueName: \"kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.778638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.778487 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.778748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.778708 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.778837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.778801 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.779288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.779260 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.779805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.779538 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.780353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.780315 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.782219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.782191 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.782535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.782512 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.791878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.791845 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p897\" (UniqueName: \"kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897\") pod \"distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:46.877169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:46.877128 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:47.073780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.073235 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 31 23:12:47.073780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.073309 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 31 23:12:47.146387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.146350 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:47.271390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.270102 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m"] Oct 31 23:12:47.373056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.373010 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" event={"ID":"3142d597-b170-4c68-8b9d-869fb8bda3b8","Type":"ContainerStarted","Data":"d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795"} Oct 31 23:12:47.373212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.373071 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" event={"ID":"3142d597-b170-4c68-8b9d-869fb8bda3b8","Type":"ContainerStarted","Data":"7f986c5490b9fdba00312659f292ed6187d497cccb46099e4347bb78e45f83de"} Oct 31 23:12:47.399341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.399287 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podStartSLOduration=1.399270514 podStartE2EDuration="1.399270514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:47.393489705 +0000 UTC m=+1393.623044130" watchObservedRunningTime="2026-10-31 23:12:47.399270514 +0000 UTC m=+1393.628824925" Oct 31 23:12:47.816596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.816510 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:47.883022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:47.882984 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:48.081150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:48.081047 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:48.081724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:48.081694 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:48.082464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:48.082443 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:12:48.464842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:48.463223 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:49.218578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.218501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:49.879756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.879725 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5"] Oct 31 23:12:49.884376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.884346 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:49.888524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.888476 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:12:49.889770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.889741 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:12:49.889770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.889748 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:12:49.889770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.889766 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:12:49.893021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.892999 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5"] Oct 31 23:12:49.932063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.918737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:49.947659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.947572 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:49.947860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.947776 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:49.947860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:49.947833 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kz6\" (UniqueName: \"kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.049010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.048973 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kz6\" (UniqueName: \"kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.049221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.049140 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.049312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.049288 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.050055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.050030 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.051727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.051703 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.061666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.061644 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kz6\" (UniqueName: \"kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6\") pod \"maas-api-revoke-sub-d015b7c9-chmr5\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.203632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.203537 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:50.501862 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:50.501832 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884c397b_4a5e_4f15_8eef_f77a866bc2ae.slice/crio-68b8cc93bfce87694934fdd0bbe19f7a8705417de21d5491b7c231a8ffa2bcd9 WatchSource:0}: Error finding container 68b8cc93bfce87694934fdd0bbe19f7a8705417de21d5491b7c231a8ffa2bcd9: Status 404 returned error can't find the container with id 68b8cc93bfce87694934fdd0bbe19f7a8705417de21d5491b7c231a8ffa2bcd9 Oct 31 23:12:50.502179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.502127 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5"] Oct 31 23:12:50.505428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:50.505380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:51.302598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:51.302527 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:51.413855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:51.413813 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" event={"ID":"884c397b-4a5e-4f15-8eef-f77a866bc2ae","Type":"ContainerStarted","Data":"a13fea42703a17a3195408a68a6ea21298ebe23c4626ea8ba78a56f66d27787b"} Oct 31 23:12:51.414111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:51.413932 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" event={"ID":"884c397b-4a5e-4f15-8eef-f77a866bc2ae","Type":"ContainerStarted","Data":"68b8cc93bfce87694934fdd0bbe19f7a8705417de21d5491b7c231a8ffa2bcd9"} Oct 31 23:12:51.440218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:51.440159 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" podStartSLOduration=2.440140467 podStartE2EDuration="2.440140467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:12:51.435856754 +0000 UTC m=+1397.665411165" watchObservedRunningTime="2026-10-31 23:12:51.440140467 +0000 UTC m=+1397.669694882" Oct 31 23:12:51.469337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:51.469301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:51.604585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:51.604472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:52.124605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:52.124512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:52.427895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:52.427818 5043 generic.go:356] "Generic (PLEG): container finished" podID="884c397b-4a5e-4f15-8eef-f77a866bc2ae" containerID="a13fea42703a17a3195408a68a6ea21298ebe23c4626ea8ba78a56f66d27787b" exitCode=0 Oct 31 23:12:52.428041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:52.427898 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" event={"ID":"884c397b-4a5e-4f15-8eef-f77a866bc2ae","Type":"ContainerDied","Data":"a13fea42703a17a3195408a68a6ea21298ebe23c4626ea8ba78a56f66d27787b"} Oct 31 23:12:52.913871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:52.913838 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:54.908026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:54.586307 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:55.159637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.156688 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.249769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.249736 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:55.461235 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:12:55.458151 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.437s" Oct 31 23:12:55.461235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.458198 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl"] Oct 31 23:12:55.461235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.459158 5043 cpu_manager.go:414] "containerMap: removing container" podUID="884c397b-4a5e-4f15-8eef-f77a866bc2ae" containerName="revoke-keys" Oct 31 23:12:55.461235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.459178 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884c397b-4a5e-4f15-8eef-f77a866bc2ae" containerName="revoke-keys" Oct 31 23:12:55.461235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.459371 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="884c397b-4a5e-4f15-8eef-f77a866bc2ae" containerName="revoke-keys" Oct 31 23:12:55.555138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.555104 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl"] Oct 31 23:12:55.572654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.556338 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.594578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.589614 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.621605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.612076 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6\" (UniqueName: \"kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6\") pod \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " Oct 31 23:12:55.621605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.612304 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\") pod \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " Oct 31 23:12:55.621605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.612350 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca\") pod \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " Oct 31 23:12:55.633590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.622435 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca" pod "884c397b-4a5e-4f15-8eef-f77a866bc2ae" (UID: "884c397b-4a5e-4f15-8eef-f77a866bc2ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:12:55.689585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.678161 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.720415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.720315 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:12:55.753887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.753839 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.754395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.754317 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.754633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.754613 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.754878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.754847 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.755137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.755102 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.769819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.769790 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.770224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.770200 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntw8\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.770430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.770396 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.770824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.770801 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:55.777191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.771123 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:55.850628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.769590 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.888759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.840143 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.888759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.840183 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:55.888759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.840204 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:12:55.888759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.864764 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.888759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.871635 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:55.927739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.923656 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 31 23:12:55.982445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.981839 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca" pod "884c397b-4a5e-4f15-8eef-f77a866bc2ae" (UID: "884c397b-4a5e-4f15-8eef-f77a866bc2ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:55.994448 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.020134 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\") pod \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\" (UID: \"884c397b-4a5e-4f15-8eef-f77a866bc2ae\") " Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:56.070121 5043 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/884c397b-4a5e-4f15-8eef-f77a866bc2ae/volumes/kubernetes.io~secret/maas-api-xks-ca Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.072760 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.072906 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntw8\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.072951 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.073070 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.073202 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.073278 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.073325 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.073388 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.073432 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.076617 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca" pod "884c397b-4a5e-4f15-8eef-f77a866bc2ae" (UID: "884c397b-4a5e-4f15-8eef-f77a866bc2ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.087284 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.087706 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.087935 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.088567 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.139586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.088817 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.177849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.154094 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6" pod "884c397b-4a5e-4f15-8eef-f77a866bc2ae" (UID: "884c397b-4a5e-4f15-8eef-f77a866bc2ae"). InnerVolumeSpecName "kube-api-access-n8kz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:12:56.209581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.176495 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.209581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.182129 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c397b-4a5e-4f15-8eef-f77a866bc2ae-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:56.304631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.301161 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.304631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.301448 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:12:56.304631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.303634 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8kz6\" (UniqueName: \"kubernetes.io/projected/884c397b-4a5e-4f15-8eef-f77a866bc2ae-kube-api-access-n8kz6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:12:56.514318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.514217 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.525339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.525296 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntw8\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.593491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.593405 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.620572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.594866 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token\") pod \"e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.641425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.641394 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:56.652568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.641732 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:12:56.689572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.681111 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:12:56.690574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.690534 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:56.733843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.733802 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.738609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.734345 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:56.816974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.816674 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.884387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.884285 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.898770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.898725 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.884676 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.884697 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.884624 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.899330 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.918208 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.918249 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.918267 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.918300 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:56.944574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:56.918330 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:57.038570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.030253 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:12:57.038570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.030737 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:57.340570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.302575 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:57.375570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.372882 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:57.392722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.392684 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:12:57.392991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.392958 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:57.437657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.411099 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" event={"ID":"884c397b-4a5e-4f15-8eef-f77a866bc2ae","Type":"ContainerDied","Data":"68b8cc93bfce87694934fdd0bbe19f7a8705417de21d5491b7c231a8ffa2bcd9"} Oct 31 23:12:57.437657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.411168 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b8cc93bfce87694934fdd0bbe19f7a8705417de21d5491b7c231a8ffa2bcd9" Oct 31 23:12:57.437657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.411241 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5" Oct 31 23:12:57.524191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.524149 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:12:57.524483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.524446 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:57.639833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.639716 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:12:57.917879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.899725 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:12:57.974227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.901766 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:12:57.986651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:57.918673 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.050989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.024699 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.050989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.050618 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:12:58.083577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.074973 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.385570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.379640 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.454051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.436135 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.460759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.460699 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:12:58.611570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.153901 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:58.611570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.526655 5043 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.733621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.056867 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:12:58.917893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.825296 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:58.918235 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:12:58.876574 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e8a9f3_8353_41c3_9ba1_d1adf0de151f.slice/crio-4c2f27748403961546865d8fbc0a4cd20b7cacfafa108e250e851826808704bf WatchSource:0}: Error finding container 4c2f27748403961546865d8fbc0a4cd20b7cacfafa108e250e851826808704bf: Status 404 returned error can't find the container with id 4c2f27748403961546865d8fbc0a4cd20b7cacfafa108e250e851826808704bf Oct 31 23:12:58.920423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.920381 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:12:58.920595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.920507 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:12:58.971687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.971650 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:58.973061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:58.973032 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:12:59.002996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.002755 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:12:59.038453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.037661 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.062794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.056925 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.204819 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:12:59.198369 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.244s" Oct 31 23:12:59.206309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.206282 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:12:59.259369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.259279 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.273954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.273899 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:59.274324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.274284 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:59.274579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.274554 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:12:59.425912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.425837 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.441241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.441184 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.485502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.485446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:12:59.614999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.614951 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.667009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.666985 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh"] Oct 31 23:12:59.792301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.784811 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:12:59.792301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.784866 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:12:59.792301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.785785 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.828680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.823521 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:12:59.828680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.824387 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:12:59.828680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.824478 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:12:59.845717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.834990 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:12:59.845717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.838942 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:12:59.849569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.848802 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkvl\" (UniqueName: \"kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.849569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.848868 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.849569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.848998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.860673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.856568 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh"] Oct 31 23:12:59.877868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.877846 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:12:59.910699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.910669 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:12:59.914974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.914875 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:12:59.939848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.924167 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:12:59.948015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.947990 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:12:59.951146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.951111 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkvl\" (UniqueName: \"kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.951274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.951195 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.951351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.951329 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.954020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.953997 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:12:59.981741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.969006 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw"] Oct 31 23:12:59.981741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.979495 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:12:59.984261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.984241 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw"] Oct 31 23:12:59.988462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.988438 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:12:59.989806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.989780 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:12:59.990133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.990053 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:12:59.994561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:12:59.992947 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:13:00.000576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.000021 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkvl\" (UniqueName: \"kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl\") pod \"maas-api-revoke-sub-2d559f2a-cb5fh\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:13:00.021617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.016195 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl"] Oct 31 23:13:00.032963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.031216 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:13:00.082520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.082435 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.084270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.084238 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9pk\" (UniqueName: \"kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.098481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.098451 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.200829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.200784 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.202030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.201897 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.202345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.202325 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9pk\" (UniqueName: \"kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.202521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.202501 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.208581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.208302 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.221279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.221249 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9pk\" (UniqueName: \"kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk\") pod \"maas-api-revoke-sub-affc41c5-d5qbw\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.284857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.284813 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:13:00.345399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.337823 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:00.406900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.406855 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" event={"ID":"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f","Type":"ContainerStarted","Data":"7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f"} Oct 31 23:13:00.407091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.406963 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" event={"ID":"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f","Type":"ContainerStarted","Data":"4c2f27748403961546865d8fbc0a4cd20b7cacfafa108e250e851826808704bf"} Oct 31 23:13:00.449998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.449935 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podStartSLOduration=6.449914797 podStartE2EDuration="6.449914797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:00.443938877 +0000 UTC m=+1406.673493308" watchObservedRunningTime="2026-10-31 23:13:00.449914797 +0000 UTC m=+1406.679469210" Oct 31 23:13:00.595200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.595134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:00.642621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.642509 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:13:00.664607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.664101 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh"] Oct 31 23:13:00.754337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:00.754285 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw"] Oct 31 23:13:01.445491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.444278 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" event={"ID":"9f16cb3a-edc2-461e-8367-31df7508a25a","Type":"ContainerStarted","Data":"aca3c88baebe1fd0c78139d02f31c89c325d6dc5c6b6774b09a686a9048aef74"} Oct 31 23:13:01.445491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.444345 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" event={"ID":"9f16cb3a-edc2-461e-8367-31df7508a25a","Type":"ContainerStarted","Data":"4b8beb407ca22a1ad1ec95fe53ebed4f9e2b2c79c50089b1336cba133c878fc4"} Oct 31 23:13:01.452211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.452130 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:01.476815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.476769 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" event={"ID":"8fea76e4-fb99-4653-9529-1f2c8357e7a9","Type":"ContainerStarted","Data":"df4706d9b3560b24f8d925e2deeeb5948c921ee75380b3f7bb8f8e64daca8b4d"} Oct 31 23:13:01.477115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.477090 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" event={"ID":"8fea76e4-fb99-4653-9529-1f2c8357e7a9","Type":"ContainerStarted","Data":"a1adb3458dc907e8ad97f52acb95c0b89a8be26751b4302c97861a44ac0c1365"} Oct 31 23:13:01.507445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.507398 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" podStartSLOduration=4.50738344 podStartE2EDuration="4.50738344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:01.484507011 +0000 UTC m=+1407.714061426" watchObservedRunningTime="2026-10-31 23:13:01.50738344 +0000 UTC m=+1407.736937850" Oct 31 23:13:01.508379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.508343 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" podStartSLOduration=4.508334695 podStartE2EDuration="4.508334695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:01.503120166 +0000 UTC m=+1407.732674577" watchObservedRunningTime="2026-10-31 23:13:01.508334695 +0000 UTC m=+1407.737889106" Oct 31 23:13:01.644265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.644166 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:13:01.644265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:01.644228 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 23:13:02.356317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.353677 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:02.480601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.480098 5043 generic.go:356] "Generic (PLEG): container finished" podID="9f16cb3a-edc2-461e-8367-31df7508a25a" containerID="aca3c88baebe1fd0c78139d02f31c89c325d6dc5c6b6774b09a686a9048aef74" exitCode=0 Oct 31 23:13:02.480601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.480265 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" event={"ID":"9f16cb3a-edc2-461e-8367-31df7508a25a","Type":"ContainerDied","Data":"aca3c88baebe1fd0c78139d02f31c89c325d6dc5c6b6774b09a686a9048aef74"} Oct 31 23:13:02.482525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.482494 5043 generic.go:356] "Generic (PLEG): container finished" podID="8fea76e4-fb99-4653-9529-1f2c8357e7a9" containerID="df4706d9b3560b24f8d925e2deeeb5948c921ee75380b3f7bb8f8e64daca8b4d" exitCode=0 Oct 31 23:13:02.482665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.482583 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" event={"ID":"8fea76e4-fb99-4653-9529-1f2c8357e7a9","Type":"ContainerDied","Data":"df4706d9b3560b24f8d925e2deeeb5948c921ee75380b3f7bb8f8e64daca8b4d"} Oct 31 23:13:02.642497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.642395 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:13:02.642497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:02.642471 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 23:13:03.079988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.079955 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:03.648736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.646008 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:13:03.648736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.646072 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 23:13:03.788273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.788235 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:03.791720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.791636 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:13:03.865234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.865109 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk\" (UniqueName: \"kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk\") pod \"9f16cb3a-edc2-461e-8367-31df7508a25a\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " Oct 31 23:13:03.865234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.865174 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca\") pod \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " Oct 31 23:13:03.865477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.865351 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl\" (UniqueName: \"kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl\") pod \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " Oct 31 23:13:03.865477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.865388 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca\") pod \"9f16cb3a-edc2-461e-8367-31df7508a25a\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " Oct 31 23:13:03.865659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.865637 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca\") pod \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\" (UID: \"8fea76e4-fb99-4653-9529-1f2c8357e7a9\") " Oct 31 23:13:03.865725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.865677 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca\") pod \"9f16cb3a-edc2-461e-8367-31df7508a25a\" (UID: \"9f16cb3a-edc2-461e-8367-31df7508a25a\") " Oct 31 23:13:03.866421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.866385 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca" pod "9f16cb3a-edc2-461e-8367-31df7508a25a" (UID: "9f16cb3a-edc2-461e-8367-31df7508a25a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:03.866421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.866397 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca" pod "8fea76e4-fb99-4653-9529-1f2c8357e7a9" (UID: "8fea76e4-fb99-4653-9529-1f2c8357e7a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:03.878258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.877812 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca" pod "9f16cb3a-edc2-461e-8367-31df7508a25a" (UID: "9f16cb3a-edc2-461e-8367-31df7508a25a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:03.878258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.878025 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk" pod "9f16cb3a-edc2-461e-8367-31df7508a25a" (UID: "9f16cb3a-edc2-461e-8367-31df7508a25a"). InnerVolumeSpecName "kube-api-access-qq9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:03.878258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.878187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca" pod "8fea76e4-fb99-4653-9529-1f2c8357e7a9" (UID: "8fea76e4-fb99-4653-9529-1f2c8357e7a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:03.880169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.880129 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl" pod "8fea76e4-fb99-4653-9529-1f2c8357e7a9" (UID: "8fea76e4-fb99-4653-9529-1f2c8357e7a9"). InnerVolumeSpecName "kube-api-access-9xkvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:03.968419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.968373 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:03.968685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.968664 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xkvl\" (UniqueName: \"kubernetes.io/projected/8fea76e4-fb99-4653-9529-1f2c8357e7a9-kube-api-access-9xkvl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:03.968798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.968786 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:03.968930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.968917 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea76e4-fb99-4653-9529-1f2c8357e7a9-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:03.969025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.969014 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f16cb3a-edc2-461e-8367-31df7508a25a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:03.969131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:03.969119 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9pk\" (UniqueName: \"kubernetes.io/projected/9f16cb3a-edc2-461e-8367-31df7508a25a-kube-api-access-qq9pk\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:04.006089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.006002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:04.501646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.501602 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" event={"ID":"9f16cb3a-edc2-461e-8367-31df7508a25a","Type":"ContainerDied","Data":"4b8beb407ca22a1ad1ec95fe53ebed4f9e2b2c79c50089b1336cba133c878fc4"} Oct 31 23:13:04.501826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.501672 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8beb407ca22a1ad1ec95fe53ebed4f9e2b2c79c50089b1336cba133c878fc4" Oct 31 23:13:04.501826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.501753 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw" Oct 31 23:13:04.505283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.505259 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" Oct 31 23:13:04.505399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.505375 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh" event={"ID":"8fea76e4-fb99-4653-9529-1f2c8357e7a9","Type":"ContainerDied","Data":"a1adb3458dc907e8ad97f52acb95c0b89a8be26751b4302c97861a44ac0c1365"} Oct 31 23:13:04.505450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.505424 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1adb3458dc907e8ad97f52acb95c0b89a8be26751b4302c97861a44ac0c1365" Oct 31 23:13:04.617571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.613819 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:04.670420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.670381 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:13:04.671029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:04.670994 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 23:13:05.336573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:05.323787 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:05.643926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:05.643058 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:13:05.643926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:05.643123 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 23:13:05.999391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:05.999353 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.109478 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh"] Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.110518 5043 cpu_manager.go:414] "containerMap: removing container" podUID="8fea76e4-fb99-4653-9529-1f2c8357e7a9" containerName="revoke-keys" Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.110538 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fea76e4-fb99-4653-9529-1f2c8357e7a9" containerName="revoke-keys" Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.110574 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9f16cb3a-edc2-461e-8367-31df7508a25a" containerName="revoke-keys" Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.110581 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f16cb3a-edc2-461e-8367-31df7508a25a" containerName="revoke-keys" Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.110804 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fea76e4-fb99-4653-9529-1f2c8357e7a9" containerName="revoke-keys" Oct 31 23:13:06.132576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.110825 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f16cb3a-edc2-461e-8367-31df7508a25a" containerName="revoke-keys" Oct 31 23:13:06.189568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.183280 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.189568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.188403 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:13:06.189568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.188913 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:13:06.189568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.188968 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:13:06.189816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.189633 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:13:06.194022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.193993 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh"] Oct 31 23:13:06.210723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.210684 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.210853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.210795 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.211116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.211094 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6kw\" (UniqueName: \"kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.312868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.312782 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6kw\" (UniqueName: \"kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.313043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.312870 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.313043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.312902 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.313599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.313540 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.315625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.315593 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.326358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.326329 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6kw\" (UniqueName: \"kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw\") pod \"maas-api-revoke-sub-42ffee0c-sldkh\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.503420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.503383 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:06.642244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.642203 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 23:13:06.642430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.642288 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 23:13:06.814204 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:06.814157 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac2704b_bdca_4833_9fd4_dc78917c9c03.slice/crio-b3d28249d83f209eeecc3cf44dc3bff082bd06cba2a0af3d7fb3313d3f412209 WatchSource:0}: Error finding container b3d28249d83f209eeecc3cf44dc3bff082bd06cba2a0af3d7fb3313d3f412209: Status 404 returned error can't find the container with id b3d28249d83f209eeecc3cf44dc3bff082bd06cba2a0af3d7fb3313d3f412209 Oct 31 23:13:06.815971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.815883 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh"] Oct 31 23:13:06.886905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.884664 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:06.904237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.904214 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:13:06.937564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:06.936485 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:13:07.230809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.226391 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq"] Oct 31 23:13:07.236594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.233502 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.272157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.272067 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq"] Oct 31 23:13:07.371056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.371021 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.371262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.371201 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpjs\" (UniqueName: \"kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.371354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.371265 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.473158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.473116 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.473387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.473361 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.473536 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.473504 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpjs\" (UniqueName: \"kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.474097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.474069 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.475838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.475806 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.489128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.489058 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpjs\" (UniqueName: \"kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs\") pod \"maas-api-revoke-sub-b4536121-qq6qq\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.529573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.529012 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:07.571576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.570507 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:07.627193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.621334 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" event={"ID":"9ac2704b-bdca-4833-9fd4-dc78917c9c03","Type":"ContainerStarted","Data":"d3a9872839632cf1b19122316bee5f7211ad004afa8d3e9cb32ddd9034fa34a2"} Oct 31 23:13:07.627193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.621393 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" event={"ID":"9ac2704b-bdca-4833-9fd4-dc78917c9c03","Type":"ContainerStarted","Data":"b3d28249d83f209eeecc3cf44dc3bff082bd06cba2a0af3d7fb3313d3f412209"} Oct 31 23:13:07.795235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.795177 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:13:07.796323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.796245 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:13:07.797482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.797455 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:13:07.823495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.823437 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" podStartSLOduration=1.823422817 podStartE2EDuration="1.823422817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:07.646784095 +0000 UTC m=+1413.876338509" watchObservedRunningTime="2026-10-31 23:13:07.823422817 +0000 UTC m=+1414.052977270" Oct 31 23:13:07.922461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:07.922296 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq"] Oct 31 23:13:08.383831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:08.383795 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:08.629868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:08.629833 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" event={"ID":"7ea6b8e4-d3b1-48f7-b502-a97009730d7f","Type":"ContainerStarted","Data":"ce1ef4ad441f0678c50bfc1ca65e872f59e589856843c0ec71fdced72b7a153c"} Oct 31 23:13:08.630076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:08.629889 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" event={"ID":"7ea6b8e4-d3b1-48f7-b502-a97009730d7f","Type":"ContainerStarted","Data":"270add3159e60de9632e1db679efc2c040325e5746ff599b6cedda911ad10965"} Oct 31 23:13:08.631760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:08.631726 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ac2704b-bdca-4833-9fd4-dc78917c9c03" containerID="d3a9872839632cf1b19122316bee5f7211ad004afa8d3e9cb32ddd9034fa34a2" exitCode=0 Oct 31 23:13:08.631892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:08.631812 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" event={"ID":"9ac2704b-bdca-4833-9fd4-dc78917c9c03","Type":"ContainerDied","Data":"d3a9872839632cf1b19122316bee5f7211ad004afa8d3e9cb32ddd9034fa34a2"} Oct 31 23:13:08.652869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:08.652800 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" podStartSLOduration=1.652783832 podStartE2EDuration="1.652783832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:08.649065643 +0000 UTC m=+1414.878620054" watchObservedRunningTime="2026-10-31 23:13:08.652783832 +0000 UTC m=+1414.882338243" Oct 31 23:13:09.204930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.204897 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:09.466305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.466219 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l"] Oct 31 23:13:09.470711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.470684 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.486378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.486349 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l"] Oct 31 23:13:09.526414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.526383 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87"] Oct 31 23:13:09.534601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.534505 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.537429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.537396 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87"] Oct 31 23:13:09.610090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610059 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9gvd\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610118 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610214 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610266 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610340 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610378 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh68q\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610396 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610464 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610498 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610650 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610682 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610774 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610841 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610872 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610888 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.610917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610902 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.610987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.610974 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.611034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.611022 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.633124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.633089 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx"] Oct 31 23:13:09.638684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.638658 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.646371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.646340 5043 generic.go:356] "Generic (PLEG): container finished" podID="7ea6b8e4-d3b1-48f7-b502-a97009730d7f" containerID="ce1ef4ad441f0678c50bfc1ca65e872f59e589856843c0ec71fdced72b7a153c" exitCode=0 Oct 31 23:13:09.646489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.646378 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" event={"ID":"7ea6b8e4-d3b1-48f7-b502-a97009730d7f","Type":"ContainerDied","Data":"ce1ef4ad441f0678c50bfc1ca65e872f59e589856843c0ec71fdced72b7a153c"} Oct 31 23:13:09.665891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.665861 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx"] Oct 31 23:13:09.713330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.713286 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.713497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.713355 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.713497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.713391 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.713497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.713443 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh68q\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.714066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714040 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.714529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714239 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.715694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714781 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.715694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714843 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.715694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714875 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.715694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714909 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.715694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.714952 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.719662 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.719717 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.719806 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.719836 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.719917 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720057 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720098 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gsw\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720143 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720172 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720197 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720229 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720265 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720294 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720344 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9gvd\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720416 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720467 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720506 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.721299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.720917 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.722668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.722644 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.723075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.723048 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.723171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.723061 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.723490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.723471 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.724323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.724289 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.725065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.725042 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.726363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.726301 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.726763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.726743 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.728665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.728638 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh68q\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.730310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.730263 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.731172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.731130 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.732396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.732371 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.734361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.734321 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.737854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.735966 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9gvd\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.737854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.737358 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.741050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.741008 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy\") pod \"e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.788873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.788658 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.824627 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.824752 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.824784 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.824819 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.824865 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.824903 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.825096 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gsw\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.825151 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.825213 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.826033 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.826506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.826316 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.827607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.826906 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.827607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.826943 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.827607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.827110 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.829566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.829533 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.831818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.831790 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.848207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.846149 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.849624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.849586 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gsw\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw\") pod \"e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.856607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.856372 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:09.886055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.885689 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:09.997734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.971565 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:09.997734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:09.972478 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:10.126223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.126188 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l"] Oct 31 23:13:10.131982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.131136 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca\") pod \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " Oct 31 23:13:10.131982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.131264 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca\") pod \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " Oct 31 23:13:10.131982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.131539 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw\" (UniqueName: \"kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw\") pod \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\" (UID: \"9ac2704b-bdca-4833-9fd4-dc78917c9c03\") " Oct 31 23:13:10.139517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.139491 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca" pod "9ac2704b-bdca-4833-9fd4-dc78917c9c03" (UID: "9ac2704b-bdca-4833-9fd4-dc78917c9c03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:10.140450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.140416 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.140578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.140501 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.141407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.141042 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw" pod "9ac2704b-bdca-4833-9fd4-dc78917c9c03" (UID: "9ac2704b-bdca-4833-9fd4-dc78917c9c03"). InnerVolumeSpecName "kube-api-access-cj6kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:10.141407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.141104 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca" pod "9ac2704b-bdca-4833-9fd4-dc78917c9c03" (UID: "9ac2704b-bdca-4833-9fd4-dc78917c9c03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:10.142724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.142637 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.193766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.193742 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87"] Oct 31 23:13:10.194316 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:10.194290 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842a6154_7c20_43e2_bc07_03a6cce2d71b.slice/crio-c6b6bb3836845ad3f4da9ac051230adc0f246617d8f784fc7c6e92100bbd2ece WatchSource:0}: Error finding container c6b6bb3836845ad3f4da9ac051230adc0f246617d8f784fc7c6e92100bbd2ece: Status 404 returned error can't find the container with id c6b6bb3836845ad3f4da9ac051230adc0f246617d8f784fc7c6e92100bbd2ece Oct 31 23:13:10.200891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.200854 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.201038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.200976 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.201038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.201027 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.238408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.238388 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6kw\" (UniqueName: \"kubernetes.io/projected/9ac2704b-bdca-4833-9fd4-dc78917c9c03-kube-api-access-cj6kw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:10.238566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.238533 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:10.239232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.239215 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac2704b-bdca-4833-9fd4-dc78917c9c03-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:10.385864 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:10.385204 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a79e0dc_3d9b_4309_af30_92c2d76e8c41.slice/crio-e8522cba4052bf65eca0e4b240e41da55de974c165aece0b14844b7f3320b034 WatchSource:0}: Error finding container e8522cba4052bf65eca0e4b240e41da55de974c165aece0b14844b7f3320b034: Status 404 returned error can't find the container with id e8522cba4052bf65eca0e4b240e41da55de974c165aece0b14844b7f3320b034 Oct 31 23:13:10.386629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.386602 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx"] Oct 31 23:13:10.387382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.387351 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.387442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.387426 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.387479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.387461 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:10.476860 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:13:10.476798 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:13:10.693091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.693013 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" event={"ID":"47d31628-1d5e-4601-9b3f-3b514fed875e","Type":"ContainerStarted","Data":"d92b844ec596fa77e27176e042eca461aad29ad582463d39277820c3fc5f9018"} Oct 31 23:13:10.693266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.693090 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" event={"ID":"47d31628-1d5e-4601-9b3f-3b514fed875e","Type":"ContainerStarted","Data":"e6232f4ee35c096c18f3e24bc8dcaf893c3166b6cf3e0615c2e90c410abeba1d"} Oct 31 23:13:10.695363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.695321 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" event={"ID":"9ac2704b-bdca-4833-9fd4-dc78917c9c03","Type":"ContainerDied","Data":"b3d28249d83f209eeecc3cf44dc3bff082bd06cba2a0af3d7fb3313d3f412209"} Oct 31 23:13:10.695471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.695370 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d28249d83f209eeecc3cf44dc3bff082bd06cba2a0af3d7fb3313d3f412209" Oct 31 23:13:10.695471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.695327 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh" Oct 31 23:13:10.697744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.697716 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" event={"ID":"9a79e0dc-3d9b-4309-af30-92c2d76e8c41","Type":"ContainerStarted","Data":"37102e32d4a7f763663cdf7c0120f8d4a6ac2dd5f783fab9ffd21b1f5b79ae2e"} Oct 31 23:13:10.697844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.697763 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" event={"ID":"9a79e0dc-3d9b-4309-af30-92c2d76e8c41","Type":"ContainerStarted","Data":"e8522cba4052bf65eca0e4b240e41da55de974c165aece0b14844b7f3320b034"} Oct 31 23:13:10.700340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.700307 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" event={"ID":"842a6154-7c20-43e2-bc07-03a6cce2d71b","Type":"ContainerStarted","Data":"3d8fadd7f7cde659665f33db449490030d5933036832b9df1eab3dc50d79cd5e"} Oct 31 23:13:10.700423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.700371 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" event={"ID":"842a6154-7c20-43e2-bc07-03a6cce2d71b","Type":"ContainerStarted","Data":"c6b6bb3836845ad3f4da9ac051230adc0f246617d8f784fc7c6e92100bbd2ece"} Oct 31 23:13:10.741388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.735970 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podStartSLOduration=1.735947644 podStartE2EDuration="1.735947644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:10.728259853 +0000 UTC m=+1416.957814264" watchObservedRunningTime="2026-10-31 23:13:10.735947644 +0000 UTC m=+1416.965502056" Oct 31 23:13:10.741388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.736072 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:10.791367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.791332 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:10.806250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.806197 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podStartSLOduration=1.8061783820000001 podStartE2EDuration="1.806178382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:10.766143518 +0000 UTC m=+1416.995697930" watchObservedRunningTime="2026-10-31 23:13:10.806178382 +0000 UTC m=+1417.035732860" Oct 31 23:13:10.811194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.811100 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podStartSLOduration=1.811082727 podStartE2EDuration="1.811082727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:10.791267401 +0000 UTC m=+1417.020821812" watchObservedRunningTime="2026-10-31 23:13:10.811082727 +0000 UTC m=+1417.040637142" Oct 31 23:13:10.863340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.856973 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:10.972490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.972458 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:10.985854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:10.985823 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:11.087598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.071320 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs\" (UniqueName: \"kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs\") pod \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " Oct 31 23:13:11.087598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.074061 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca\") pod \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " Oct 31 23:13:11.087598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.074191 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca\") pod \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\" (UID: \"7ea6b8e4-d3b1-48f7-b502-a97009730d7f\") " Oct 31 23:13:11.087598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.082066 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca" pod "7ea6b8e4-d3b1-48f7-b502-a97009730d7f" (UID: "7ea6b8e4-d3b1-48f7-b502-a97009730d7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:11.097116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.097068 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs" pod "7ea6b8e4-d3b1-48f7-b502-a97009730d7f" (UID: "7ea6b8e4-d3b1-48f7-b502-a97009730d7f"). InnerVolumeSpecName "kube-api-access-ggpjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:11.103310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.103280 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca" pod "7ea6b8e4-d3b1-48f7-b502-a97009730d7f" (UID: "7ea6b8e4-d3b1-48f7-b502-a97009730d7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:11.178514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.178473 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpjs\" (UniqueName: \"kubernetes.io/projected/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-kube-api-access-ggpjs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:11.178751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.178731 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:11.178871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.178856 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ea6b8e4-d3b1-48f7-b502-a97009730d7f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:11.544503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.544464 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:11.912775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.879676 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:11.912775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.879749 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:11.912775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.883454 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:11.912775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:11.883508 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:12.006660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.006615 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:12.006926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.006898 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:12.077672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.077623 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" event={"ID":"7ea6b8e4-d3b1-48f7-b502-a97009730d7f","Type":"ContainerDied","Data":"270add3159e60de9632e1db679efc2c040325e5746ff599b6cedda911ad10965"} Oct 31 23:13:12.077953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.077931 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="270add3159e60de9632e1db679efc2c040325e5746ff599b6cedda911ad10965" Oct 31 23:13:12.078161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.078147 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq" Oct 31 23:13:12.686571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.673620 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 23:13:12.794337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.794291 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:12.796886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.796842 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:12.874578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.868939 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:12.874578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.868997 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:12.980993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.980900 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:12.981249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:12.981217 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:13.793570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.790652 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:13.793570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.790720 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:13.864568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.863997 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:13.864568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.864058 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:13.932717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.932672 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:13.937367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.937334 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:13.974065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.973655 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:13.974065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:13.973715 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:14.010025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.009479 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:14.382615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.382566 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:14.419567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.406584 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:14.437805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.437760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:14.617571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.592764 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:14.642845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.642716 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:14.662419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.662332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:14.790000 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.789961 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:14.790258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.790228 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:14.859038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.859003 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:14.859615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.859580 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:14.990626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.990173 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:14.990626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:14.990518 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:13:15.009878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.004680 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.009878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.005075 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:13:15.035570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.033009 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.046686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.046626 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.084583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.066631 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.084583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.072131 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:15.084583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.072177 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:15.127687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.127653 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:15.472753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.472697 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.520571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.520259 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:13:15.612184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.611719 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.894679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.894645 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:15.895254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.895219 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:15.911032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.908466 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:15.911730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.911698 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:15.912249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.912217 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:15.956571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.954956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:15.977575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.977434 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:15.977575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:15.977499 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:16.030601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.025064 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.030601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.025629 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:16.031039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.031005 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.040461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.040421 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.093196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.044352 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.104912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.047613 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.104912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.070524 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.187631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.150221 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.187631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.180188 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:13:16.187631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.180303 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:13:16.187631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.180359 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:13:16.276572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.273575 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:16.489623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.479676 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:16.561572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.560528 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.613582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.605470 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:13:16.823742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.821840 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:16.823742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.821899 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:16.868236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.865183 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:13:16.868236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.867835 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:16.868236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.867886 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:16.926598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.918740 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:16.926598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.919136 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:13:17.001560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.984154 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:17.001560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:16.984212 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:17.031585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.027833 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.031585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.028274 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.031585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.028664 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:13:17.031585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.031239 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.101359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.094736 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:13:17.101359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.100321 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:13:17.101359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.100379 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:13:17.117920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.117863 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.160607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.160568 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:17.161490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.161434 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:13:17.288130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.288086 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.288714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.288690 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:13:17.457178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.457047 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:17.524759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.501166 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.529696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.518987 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:13:17.561161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.561107 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.572888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.572843 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:13:17.690839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.690717 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:17.790573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.780831 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:13:17.791470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.791433 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:17.791710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.791681 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:17.997367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:17.997312 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:18.052739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.036203 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:18.052739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.036274 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:18.168130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.168080 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:13:18.238570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.235024 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:18.276569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.251388 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:18.330995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.330874 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:18.427609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.333436 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:18.588659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.385191 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:18.588898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.588872 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:18.589027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.536308 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:18.589222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.507563 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:18.589377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.545961 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:18.589807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.587705 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:18.731652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.729157 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:18.782706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.782638 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:18.794039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.794005 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:13:18.829631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.810297 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:18.829943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.829904 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:18.865878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.865790 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:18.866118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.866088 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:18.917573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.912654 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:13:18.989153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.988709 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:18.989153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:18.988772 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:19.189372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.189270 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.264571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.243710 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.267141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.267099 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:19.298570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.284306 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:19.439175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.439136 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.603408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.597540 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.603408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.597991 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:19.603408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.598283 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:19.603408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.598538 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:19.603408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.599328 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:13:19.729575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.729353 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.815582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.809685 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.815582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.810345 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:13:19.907837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.817276 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:19.912307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.912265 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:19.958654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.925150 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:19.958654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.925204 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:19.958654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.925272 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:19.958654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:19.925615 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:13:20.031614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.021460 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.031614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.021895 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:20.031614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.021936 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:20.235631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.234796 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:13:20.303671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.289705 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.303671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.299657 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:20.394577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.334733 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:13:20.394577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.356691 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:13:20.444811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.444761 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.662658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.636674 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.663323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.636637 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:20.663618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.663588 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:20.748569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.681569 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.758576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.749284 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.799220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.799183 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:13:20.800617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.800585 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:13:20.841534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.813317 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.951761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.951660 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:13:20.951995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.951969 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:13:20.952107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.951755 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:20.984331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:20.984293 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:13:21.045060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.045012 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:13:21.046743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.046710 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 31 23:13:21.306720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.306654 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.563658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.260636 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.621044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.441007 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:21.635610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.442987 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:21.872688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.518133 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:13:21.872688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.718156 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:21.872688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.805207 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:21.876586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.876246 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:21.876586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:21.876303 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.049571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.037662 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.049571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.038105 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.122654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.118667 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:22.122654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.119093 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.122654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.121037 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:22.254400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.253845 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.853469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.852798 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:22.882323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.882277 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.894771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.894717 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:22.898571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.895028 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.924296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.895080 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:22.925572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.895833 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:22.925800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.925746 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.926520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.895975 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.927615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.896868 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:22.957357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.896942 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:22.959596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:22.896917 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.014882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.014844 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:23.078577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.078261 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.078782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.078571 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.136221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.136121 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.165011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.149411 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:23.209359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.209319 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:23.209703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.209659 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.262831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.262774 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:23.263169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.263138 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.318726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.318668 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:23.446626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.421516 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:23.692571 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:13:23.625154 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.464s" Oct 31 23:13:23.692571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.681987 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:13:23.863658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.862978 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:23.901182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.901141 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:23.947608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.932731 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:23.947608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.937644 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:23.947608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.939169 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:13:23.996573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.955258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:24.018598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.962719 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:24.147625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.988614 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:24.147625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:23.989784 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.147625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.081677 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.147625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.081712 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.147625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.133853 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.151561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.149175 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.248002 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.247704 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.293615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.291166 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:24.293615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.291231 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.418643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.398101 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:24.418643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.398161 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.418643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.407585 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:24.418643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.407638 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:24.444879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.444849 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:13:24.461067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.461040 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:13:24.477530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.477502 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:13:24.480895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.480875 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:13:24.481324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.481302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:24.488815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.483471 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:13:24.488815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.485129 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:13:24.488815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.486876 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:24.492882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.491293 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:24.492882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.491372 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:13:24.492882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.491406 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:13:24.492882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.491528 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:13:24.492882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.491583 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:13:24.496600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.494059 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:13:24.514562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.510143 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:13:24.521688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.521659 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:13:24.696353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.693641 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:24.859062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.857983 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:24.994618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:24.978029 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:25.366625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:25.357957 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:25.500992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:25.500968 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:13:25.514656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:25.514623 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:25.514936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:25.514898 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:25.650642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:25.612613 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:26.348571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:26.341869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:26.487369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:26.483955 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:26.487369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:26.486668 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:13:27.156011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.155971 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:27.661675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.661640 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq"] Oct 31 23:13:27.662676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.662647 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9ac2704b-bdca-4833-9fd4-dc78917c9c03" containerName="revoke-keys" Oct 31 23:13:27.662676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.662671 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac2704b-bdca-4833-9fd4-dc78917c9c03" containerName="revoke-keys" Oct 31 23:13:27.662853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.662748 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7ea6b8e4-d3b1-48f7-b502-a97009730d7f" containerName="revoke-keys" Oct 31 23:13:27.662853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.662757 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea6b8e4-d3b1-48f7-b502-a97009730d7f" containerName="revoke-keys" Oct 31 23:13:27.662943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.662929 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea6b8e4-d3b1-48f7-b502-a97009730d7f" containerName="revoke-keys" Oct 31 23:13:27.662978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.662955 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac2704b-bdca-4833-9fd4-dc78917c9c03" containerName="revoke-keys" Oct 31 23:13:27.667847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.667827 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.670619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.670598 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq"] Oct 31 23:13:27.672119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.672087 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:13:27.672308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.672287 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:13:27.673644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.673622 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:13:27.673747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.673690 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:13:27.725302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.725271 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.725470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.725444 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.725517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.725495 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29lcb\" (UniqueName: \"kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.827009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.826973 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.827184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.827021 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29lcb\" (UniqueName: \"kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.827265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.827251 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.827999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.827972 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.829602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.829575 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.839638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.839612 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29lcb\" (UniqueName: \"kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb\") pod \"maas-api-revoke-sub-4cc6360c-pshrq\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:27.916491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.916412 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:27.987478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:27.987440 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:28.328635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.328603 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq"] Oct 31 23:13:28.330238 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:28.330205 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4dfdd2_48b9_4613_aee0_18e7090394d8.slice/crio-b8d1b1e1762b537a03e429f1a1046a8804ac3171976470b5846053fd4d218251 WatchSource:0}: Error finding container b8d1b1e1762b537a03e429f1a1046a8804ac3171976470b5846053fd4d218251: Status 404 returned error can't find the container with id b8d1b1e1762b537a03e429f1a1046a8804ac3171976470b5846053fd4d218251 Oct 31 23:13:28.583655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.583584 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2"] Oct 31 23:13:28.589582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.589558 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.600487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.600463 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2"] Oct 31 23:13:28.639594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.638024 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:28.663641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.652259 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.663641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.652493 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.663641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.653992 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446vq\" (UniqueName: \"kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.757941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.756071 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446vq\" (UniqueName: \"kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.757941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.756287 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.757941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.756485 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.757941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.757113 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.772281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.771598 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.772888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.772859 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446vq\" (UniqueName: \"kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq\") pod \"maas-api-revoke-sub-9f3fc202-8fsp2\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:28.907368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:28.907292 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:29.084786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.084748 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" event={"ID":"3f4dfdd2-48b9-4613-aee0-18e7090394d8","Type":"ContainerStarted","Data":"2fef461a9c8e9ead1ff06ed758f90dbdda6058511bda170ca4efed4e112edde8"} Oct 31 23:13:29.085059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.085034 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" event={"ID":"3f4dfdd2-48b9-4613-aee0-18e7090394d8","Type":"ContainerStarted","Data":"b8d1b1e1762b537a03e429f1a1046a8804ac3171976470b5846053fd4d218251"} Oct 31 23:13:29.112330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.112282 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" podStartSLOduration=2.112268845 podStartE2EDuration="2.112268845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:29.108348885 +0000 UTC m=+1435.337903299" watchObservedRunningTime="2026-10-31 23:13:29.112268845 +0000 UTC m=+1435.341823256" Oct 31 23:13:29.204239 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:29.204210 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48eab729_858d_47b4_a23a_a4cb42abfa40.slice/crio-8a27817cbd99bdd44288fece4cca3b286f56043c3fc9980f52524e81666fbac2 WatchSource:0}: Error finding container 8a27817cbd99bdd44288fece4cca3b286f56043c3fc9980f52524e81666fbac2: Status 404 returned error can't find the container with id 8a27817cbd99bdd44288fece4cca3b286f56043c3fc9980f52524e81666fbac2 Oct 31 23:13:29.204935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.204909 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2"] Oct 31 23:13:29.297370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.297331 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw"] Oct 31 23:13:29.303565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.303520 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.309994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.309963 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw"] Oct 31 23:13:29.367394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.367357 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.367588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.367424 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpsd\" (UniqueName: \"kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.367703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.367671 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.463514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.463443 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:29.469276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.469243 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.492887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.478619 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.492887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.481976 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpsd\" (UniqueName: \"kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.492887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.482222 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.499476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.499442 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.509675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.507514 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpsd\" (UniqueName: \"kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd\") pod \"maas-api-revoke-sub-3f590727-cgmrw\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.621117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.621088 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:29.922618 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:29.922580 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe45fe22_12e1_43fe_8192_ec1028054740.slice/crio-81fd63070caed0c717199e82d25393cc24eadacd7511efbbaa02d2060fc86c0c WatchSource:0}: Error finding container 81fd63070caed0c717199e82d25393cc24eadacd7511efbbaa02d2060fc86c0c: Status 404 returned error can't find the container with id 81fd63070caed0c717199e82d25393cc24eadacd7511efbbaa02d2060fc86c0c Oct 31 23:13:29.923326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.923299 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw"] Oct 31 23:13:29.992009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.991980 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v"] Oct 31 23:13:29.997874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:29.997847 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.027411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.027378 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v"] Oct 31 23:13:30.090919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.090873 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.091092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.091068 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.091150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.091113 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8dl\" (UniqueName: \"kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.096784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.096711 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" event={"ID":"48eab729-858d-47b4-a23a-a4cb42abfa40","Type":"ContainerStarted","Data":"92ce79b766b91216e54278ebc6a81fdbc9294e59c44595b801fb22e226e16d7a"} Oct 31 23:13:30.096784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.096765 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" event={"ID":"48eab729-858d-47b4-a23a-a4cb42abfa40","Type":"ContainerStarted","Data":"8a27817cbd99bdd44288fece4cca3b286f56043c3fc9980f52524e81666fbac2"} Oct 31 23:13:30.098324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.098303 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" event={"ID":"be45fe22-12e1-43fe-8192-ec1028054740","Type":"ContainerStarted","Data":"81fd63070caed0c717199e82d25393cc24eadacd7511efbbaa02d2060fc86c0c"} Oct 31 23:13:30.100011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.099989 5043 generic.go:356] "Generic (PLEG): container finished" podID="3f4dfdd2-48b9-4613-aee0-18e7090394d8" containerID="2fef461a9c8e9ead1ff06ed758f90dbdda6058511bda170ca4efed4e112edde8" exitCode=0 Oct 31 23:13:30.100111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.100022 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" event={"ID":"3f4dfdd2-48b9-4613-aee0-18e7090394d8","Type":"ContainerDied","Data":"2fef461a9c8e9ead1ff06ed758f90dbdda6058511bda170ca4efed4e112edde8"} Oct 31 23:13:30.122379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.122340 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" podStartSLOduration=2.122328674 podStartE2EDuration="2.122328674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:30.115039155 +0000 UTC m=+1436.344593569" watchObservedRunningTime="2026-10-31 23:13:30.122328674 +0000 UTC m=+1436.351883084" Oct 31 23:13:30.192391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.192352 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.192517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.192401 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8dl\" (UniqueName: \"kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.192639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.192623 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.193117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.193088 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.194956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.194934 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.206989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.206948 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8dl\" (UniqueName: \"kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl\") pod \"maas-api-revoke-sub-13bbe0d3-hb58v\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.309304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.309267 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:30.332897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.332860 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:30.653893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:30.653859 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v"] Oct 31 23:13:31.104789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.104747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:31.114581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.114519 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" event={"ID":"7aff63cd-52bc-42ec-993f-969f8d7339b5","Type":"ContainerStarted","Data":"357e746c74a73eece1a6bd4bfdb496afcb437434eb140cb40b92d28ff0d2d8de"} Oct 31 23:13:31.114858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.114836 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" event={"ID":"7aff63cd-52bc-42ec-993f-969f8d7339b5","Type":"ContainerStarted","Data":"a7923dec673f193583fa765a5af1c86e74be310a1c3f5c71558cef6454a58ed6"} Oct 31 23:13:31.118328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.118300 5043 generic.go:356] "Generic (PLEG): container finished" podID="48eab729-858d-47b4-a23a-a4cb42abfa40" containerID="92ce79b766b91216e54278ebc6a81fdbc9294e59c44595b801fb22e226e16d7a" exitCode=0 Oct 31 23:13:31.118667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.118643 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" event={"ID":"48eab729-858d-47b4-a23a-a4cb42abfa40","Type":"ContainerDied","Data":"92ce79b766b91216e54278ebc6a81fdbc9294e59c44595b801fb22e226e16d7a"} Oct 31 23:13:31.121826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.121800 5043 generic.go:356] "Generic (PLEG): container finished" podID="be45fe22-12e1-43fe-8192-ec1028054740" containerID="30638dc7a7bc664225976b15d39c9a7cf87dfbf38f6b467f96c16f00f4994847" exitCode=0 Oct 31 23:13:31.137399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.132709 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" event={"ID":"be45fe22-12e1-43fe-8192-ec1028054740","Type":"ContainerDied","Data":"30638dc7a7bc664225976b15d39c9a7cf87dfbf38f6b467f96c16f00f4994847"} Oct 31 23:13:31.147360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.147301 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" podStartSLOduration=2.147285569 podStartE2EDuration="2.147285569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:31.139211481 +0000 UTC m=+1437.368765893" watchObservedRunningTime="2026-10-31 23:13:31.147285569 +0000 UTC m=+1437.376839979" Oct 31 23:13:31.402196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.402170 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:31.516148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.516056 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca\") pod \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " Oct 31 23:13:31.516148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.516136 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb\" (UniqueName: \"kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb\") pod \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " Oct 31 23:13:31.516367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.516344 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca\") pod \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\" (UID: \"3f4dfdd2-48b9-4613-aee0-18e7090394d8\") " Oct 31 23:13:31.516927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.516879 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca" pod "3f4dfdd2-48b9-4613-aee0-18e7090394d8" (UID: "3f4dfdd2-48b9-4613-aee0-18e7090394d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:31.517192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.517171 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:31.518368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.518337 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca" pod "3f4dfdd2-48b9-4613-aee0-18e7090394d8" (UID: "3f4dfdd2-48b9-4613-aee0-18e7090394d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:31.518767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.518743 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb" pod "3f4dfdd2-48b9-4613-aee0-18e7090394d8" (UID: "3f4dfdd2-48b9-4613-aee0-18e7090394d8"). InnerVolumeSpecName "kube-api-access-29lcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:31.619147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.619117 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfdd2-48b9-4613-aee0-18e7090394d8-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:31.619147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.619147 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29lcb\" (UniqueName: \"kubernetes.io/projected/3f4dfdd2-48b9-4613-aee0-18e7090394d8-kube-api-access-29lcb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.004637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:31.999360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:32.145317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.145229 5043 generic.go:356] "Generic (PLEG): container finished" podID="7aff63cd-52bc-42ec-993f-969f8d7339b5" containerID="357e746c74a73eece1a6bd4bfdb496afcb437434eb140cb40b92d28ff0d2d8de" exitCode=0 Oct 31 23:13:32.145789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.145313 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" event={"ID":"7aff63cd-52bc-42ec-993f-969f8d7339b5","Type":"ContainerDied","Data":"357e746c74a73eece1a6bd4bfdb496afcb437434eb140cb40b92d28ff0d2d8de"} Oct 31 23:13:32.147160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.147132 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" Oct 31 23:13:32.147283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.147180 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq" event={"ID":"3f4dfdd2-48b9-4613-aee0-18e7090394d8","Type":"ContainerDied","Data":"b8d1b1e1762b537a03e429f1a1046a8804ac3171976470b5846053fd4d218251"} Oct 31 23:13:32.147283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.147254 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d1b1e1762b537a03e429f1a1046a8804ac3171976470b5846053fd4d218251" Oct 31 23:13:32.436854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.436823 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:32.441196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.441171 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:32.534832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.534794 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca\") pod \"be45fe22-12e1-43fe-8192-ec1028054740\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " Oct 31 23:13:32.535031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.534921 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd\" (UniqueName: \"kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd\") pod \"be45fe22-12e1-43fe-8192-ec1028054740\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " Oct 31 23:13:32.535031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.534995 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca\") pod \"be45fe22-12e1-43fe-8192-ec1028054740\" (UID: \"be45fe22-12e1-43fe-8192-ec1028054740\") " Oct 31 23:13:32.535195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.535177 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca\") pod \"48eab729-858d-47b4-a23a-a4cb42abfa40\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " Oct 31 23:13:32.535333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.535317 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq\" (UniqueName: \"kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq\") pod \"48eab729-858d-47b4-a23a-a4cb42abfa40\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " Oct 31 23:13:32.535394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.535360 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca\") pod \"48eab729-858d-47b4-a23a-a4cb42abfa40\" (UID: \"48eab729-858d-47b4-a23a-a4cb42abfa40\") " Oct 31 23:13:32.536285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.536189 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca" pod "be45fe22-12e1-43fe-8192-ec1028054740" (UID: "be45fe22-12e1-43fe-8192-ec1028054740"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:32.536412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.536348 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca" pod "48eab729-858d-47b4-a23a-a4cb42abfa40" (UID: "48eab729-858d-47b4-a23a-a4cb42abfa40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:32.540047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.540016 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd" pod "be45fe22-12e1-43fe-8192-ec1028054740" (UID: "be45fe22-12e1-43fe-8192-ec1028054740"). InnerVolumeSpecName "kube-api-access-xwpsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:32.542649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.542618 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq" pod "48eab729-858d-47b4-a23a-a4cb42abfa40" (UID: "48eab729-858d-47b4-a23a-a4cb42abfa40"). InnerVolumeSpecName "kube-api-access-446vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:32.543058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.543033 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca" pod "48eab729-858d-47b4-a23a-a4cb42abfa40" (UID: "48eab729-858d-47b4-a23a-a4cb42abfa40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:32.543695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.543665 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca" pod "be45fe22-12e1-43fe-8192-ec1028054740" (UID: "be45fe22-12e1-43fe-8192-ec1028054740"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:32.637195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.637159 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwpsd\" (UniqueName: \"kubernetes.io/projected/be45fe22-12e1-43fe-8192-ec1028054740-kube-api-access-xwpsd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.637195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.637190 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.637398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.637206 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.637398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.637219 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-446vq\" (UniqueName: \"kubernetes.io/projected/48eab729-858d-47b4-a23a-a4cb42abfa40-kube-api-access-446vq\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.637398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.637232 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48eab729-858d-47b4-a23a-a4cb42abfa40-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.637398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.637243 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be45fe22-12e1-43fe-8192-ec1028054740-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:32.793186 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:32.793150 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:33.158380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.158301 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" Oct 31 23:13:33.158380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.158340 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2" event={"ID":"48eab729-858d-47b4-a23a-a4cb42abfa40","Type":"ContainerDied","Data":"8a27817cbd99bdd44288fece4cca3b286f56043c3fc9980f52524e81666fbac2"} Oct 31 23:13:33.158964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.158396 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a27817cbd99bdd44288fece4cca3b286f56043c3fc9980f52524e81666fbac2" Oct 31 23:13:33.159920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.159898 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" Oct 31 23:13:33.160048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.159901 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw" event={"ID":"be45fe22-12e1-43fe-8192-ec1028054740","Type":"ContainerDied","Data":"81fd63070caed0c717199e82d25393cc24eadacd7511efbbaa02d2060fc86c0c"} Oct 31 23:13:33.160048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.160010 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81fd63070caed0c717199e82d25393cc24eadacd7511efbbaa02d2060fc86c0c" Oct 31 23:13:33.381804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.381777 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:33.419055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.418724 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:33.453919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.453521 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca\") pod \"7aff63cd-52bc-42ec-993f-969f8d7339b5\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " Oct 31 23:13:33.453919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.453693 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca\") pod \"7aff63cd-52bc-42ec-993f-969f8d7339b5\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " Oct 31 23:13:33.453919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.453861 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl\" (UniqueName: \"kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl\") pod \"7aff63cd-52bc-42ec-993f-969f8d7339b5\" (UID: \"7aff63cd-52bc-42ec-993f-969f8d7339b5\") " Oct 31 23:13:33.466581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.456107 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca" pod "7aff63cd-52bc-42ec-993f-969f8d7339b5" (UID: "7aff63cd-52bc-42ec-993f-969f8d7339b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:33.466581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.456467 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:33.484690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.481219 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca" pod "7aff63cd-52bc-42ec-993f-969f8d7339b5" (UID: "7aff63cd-52bc-42ec-993f-969f8d7339b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:33.484690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.484639 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl" pod "7aff63cd-52bc-42ec-993f-969f8d7339b5" (UID: "7aff63cd-52bc-42ec-993f-969f8d7339b5"). InnerVolumeSpecName "kube-api-access-xm8dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:33.558015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.557982 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aff63cd-52bc-42ec-993f-969f8d7339b5-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:33.558015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:33.558017 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm8dl\" (UniqueName: \"kubernetes.io/projected/7aff63cd-52bc-42ec-993f-969f8d7339b5-kube-api-access-xm8dl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:34.103316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:34.099930 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:34.184917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:34.184832 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" event={"ID":"7aff63cd-52bc-42ec-993f-969f8d7339b5","Type":"ContainerDied","Data":"a7923dec673f193583fa765a5af1c86e74be310a1c3f5c71558cef6454a58ed6"} Oct 31 23:13:34.184917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:34.184894 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7923dec673f193583fa765a5af1c86e74be310a1c3f5c71558cef6454a58ed6" Oct 31 23:13:34.185350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:34.185013 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v" Oct 31 23:13:34.799918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:34.799879 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:35.499100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:35.499063 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:36.139123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.139083 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:36.156967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.156409 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx"] Oct 31 23:13:36.165668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.165638 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7aff63cd-52bc-42ec-993f-969f8d7339b5" containerName="revoke-keys" Oct 31 23:13:36.165976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.165949 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aff63cd-52bc-42ec-993f-969f8d7339b5" containerName="revoke-keys" Oct 31 23:13:36.166142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166129 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3f4dfdd2-48b9-4613-aee0-18e7090394d8" containerName="revoke-keys" Oct 31 23:13:36.166221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166211 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4dfdd2-48b9-4613-aee0-18e7090394d8" containerName="revoke-keys" Oct 31 23:13:36.166338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166328 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be45fe22-12e1-43fe-8192-ec1028054740" containerName="revoke-keys" Oct 31 23:13:36.166417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166405 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be45fe22-12e1-43fe-8192-ec1028054740" containerName="revoke-keys" Oct 31 23:13:36.166528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166508 5043 cpu_manager.go:414] "containerMap: removing container" podUID="48eab729-858d-47b4-a23a-a4cb42abfa40" containerName="revoke-keys" Oct 31 23:13:36.166528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166530 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48eab729-858d-47b4-a23a-a4cb42abfa40" containerName="revoke-keys" Oct 31 23:13:36.166783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166768 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aff63cd-52bc-42ec-993f-969f8d7339b5" containerName="revoke-keys" Oct 31 23:13:36.166842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166791 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4dfdd2-48b9-4613-aee0-18e7090394d8" containerName="revoke-keys" Oct 31 23:13:36.166893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166855 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be45fe22-12e1-43fe-8192-ec1028054740" containerName="revoke-keys" Oct 31 23:13:36.166893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.166870 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="48eab729-858d-47b4-a23a-a4cb42abfa40" containerName="revoke-keys" Oct 31 23:13:36.173213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.173189 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.174619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.174592 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx"] Oct 31 23:13:36.177960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.177935 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:13:36.181979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.179810 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:13:36.181979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.179848 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:13:36.181979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.179899 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:13:36.315130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.315081 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.315284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.315154 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8kwh\" (UniqueName: \"kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.315386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.315370 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.417381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.417278 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.417381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.417334 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kwh\" (UniqueName: \"kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.417645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.417460 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.418128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.418106 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.419835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.419806 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.429904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.429880 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kwh\" (UniqueName: \"kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh\") pod \"maas-api-revoke-sub-cace0450-qnjfx\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.510438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.510405 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:36.836751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.827726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:36.847506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:36.844627 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx"] Oct 31 23:13:37.231653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:37.231605 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" event={"ID":"3075e7d2-db72-431f-90c9-34fcfea0e9a9","Type":"ContainerStarted","Data":"4895b295bd8fff51fb6f4fcc2c8b9bcb381f4720616b5c860d2558454f5728fa"} Oct 31 23:13:37.232165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:37.231675 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" event={"ID":"3075e7d2-db72-431f-90c9-34fcfea0e9a9","Type":"ContainerStarted","Data":"27d9a150c895afe4efb01a57f659a9310fd672345a65a553b791aa3f60701607"} Oct 31 23:13:37.256586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:37.256509 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" podStartSLOduration=1.256491453 podStartE2EDuration="1.256491453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:37.25039893 +0000 UTC m=+1443.479953345" watchObservedRunningTime="2026-10-31 23:13:37.256491453 +0000 UTC m=+1443.486045865" Oct 31 23:13:37.437306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:37.437109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:38.095212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:38.095181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:38.244948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:38.244896 5043 generic.go:356] "Generic (PLEG): container finished" podID="3075e7d2-db72-431f-90c9-34fcfea0e9a9" containerID="4895b295bd8fff51fb6f4fcc2c8b9bcb381f4720616b5c860d2558454f5728fa" exitCode=0 Oct 31 23:13:38.245138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:38.244969 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" event={"ID":"3075e7d2-db72-431f-90c9-34fcfea0e9a9","Type":"ContainerDied","Data":"4895b295bd8fff51fb6f4fcc2c8b9bcb381f4720616b5c860d2558454f5728fa"} Oct 31 23:13:38.779764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:38.779722 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:39.446253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.446222 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:39.589104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.588030 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca\") pod \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " Oct 31 23:13:39.589104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.588197 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca\") pod \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " Oct 31 23:13:39.589104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.588368 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh\" (UniqueName: \"kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh\") pod \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\" (UID: \"3075e7d2-db72-431f-90c9-34fcfea0e9a9\") " Oct 31 23:13:39.589104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.588592 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca" pod "3075e7d2-db72-431f-90c9-34fcfea0e9a9" (UID: "3075e7d2-db72-431f-90c9-34fcfea0e9a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:39.589437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.589111 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:39.599567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.590678 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh" pod "3075e7d2-db72-431f-90c9-34fcfea0e9a9" (UID: "3075e7d2-db72-431f-90c9-34fcfea0e9a9"). InnerVolumeSpecName "kube-api-access-f8kwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:39.605562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.600651 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca" pod "3075e7d2-db72-431f-90c9-34fcfea0e9a9" (UID: "3075e7d2-db72-431f-90c9-34fcfea0e9a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:39.649015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.648980 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp"] Oct 31 23:13:39.650055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.650027 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3075e7d2-db72-431f-90c9-34fcfea0e9a9" containerName="revoke-keys" Oct 31 23:13:39.650055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.650054 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3075e7d2-db72-431f-90c9-34fcfea0e9a9" containerName="revoke-keys" Oct 31 23:13:39.650298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.650283 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3075e7d2-db72-431f-90c9-34fcfea0e9a9" containerName="revoke-keys" Oct 31 23:13:39.654959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.654927 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.678461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.678429 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp"] Oct 31 23:13:39.690947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.690911 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.690979 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691024 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691088 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691133 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691175 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691230 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691261 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691340 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b92f\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.691472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691455 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8kwh\" (UniqueName: \"kubernetes.io/projected/3075e7d2-db72-431f-90c9-34fcfea0e9a9-kube-api-access-f8kwh\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:39.691558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.691479 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3075e7d2-db72-431f-90c9-34fcfea0e9a9-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:39.721579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.718765 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf"] Oct 31 23:13:39.721579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.720857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:39.741847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.741803 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.785129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.784449 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf"] Oct 31 23:13:39.800529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800488 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.800742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800573 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.800742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800650 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.800742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800705 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.800928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800746 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.800928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800773 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.800928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800811 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.801085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800943 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.801085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.800996 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.801085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801035 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.801239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801089 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.801239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801174 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b92f\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.801347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801270 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.801347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801298 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wqp\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.801347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801329 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.801508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801423 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.801777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801749 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.801859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801803 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.801924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.801866 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.802290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.802261 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.802444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.802419 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.802571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.802488 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.804973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.804946 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.806854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.806828 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.817065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.815871 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.849917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.849832 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk"] Oct 31 23:13:39.851393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.851364 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b92f\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.854077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.854051 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:39.857063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.857041 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:39.872120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.872088 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk"] Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.903987 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjzw\" (UniqueName: \"kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904100 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904142 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904168 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wqp\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904285 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904313 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904392 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904533 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904614 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904668 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904779 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.904838 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.905934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.905804 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.906354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.906328 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.906637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.906612 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.907082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.907058 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.907279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.907256 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.907950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.907925 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.909829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.909809 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.918004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.917983 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wqp\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.927907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.927879 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token\") pod \"e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:39.976073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:39.976046 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:40.006462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.006432 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjzw\" (UniqueName: \"kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.006719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.006538 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.006819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.006800 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.007474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.007451 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.009066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.009041 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.019515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.019491 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjzw\" (UniqueName: \"kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw\") pod \"maas-api-revoke-sub-cf8c3615-x2wpk\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.111146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.111075 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:40.186666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.185348 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:40.284529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.284474 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" event={"ID":"3075e7d2-db72-431f-90c9-34fcfea0e9a9","Type":"ContainerDied","Data":"27d9a150c895afe4efb01a57f659a9310fd672345a65a553b791aa3f60701607"} Oct 31 23:13:40.284712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.284574 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d9a150c895afe4efb01a57f659a9310fd672345a65a553b791aa3f60701607" Oct 31 23:13:40.284712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.284652 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx" Oct 31 23:13:40.287088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.287062 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp"] Oct 31 23:13:40.295368 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:40.294682 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d8b699_311d_48f7_be78_2223ba357359.slice/crio-8d36526d60e14a955eb138e33b129060d2f8f0bd160c70196d5136e7d112e71e WatchSource:0}: Error finding container 8d36526d60e14a955eb138e33b129060d2f8f0bd160c70196d5136e7d112e71e: Status 404 returned error can't find the container with id 8d36526d60e14a955eb138e33b129060d2f8f0bd160c70196d5136e7d112e71e Oct 31 23:13:40.299114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.298721 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:40.299114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.298788 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:40.299114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.298815 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:40.428196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.428146 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf"] Oct 31 23:13:40.456791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.456709 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:40.457979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.457488 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:40.457979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.457589 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:13:40.510129 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:13:40.510098 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:13:40.537300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.537051 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk"] Oct 31 23:13:40.539614 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:40.539511 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0320a6d_9fd8_4704_b0f7_5db4bfc6307a.slice/crio-f28ee883dbe1fd6ba5aefd758ce04fe5d70f514e06714685b9af71ccc86d8630 WatchSource:0}: Error finding container f28ee883dbe1fd6ba5aefd758ce04fe5d70f514e06714685b9af71ccc86d8630: Status 404 returned error can't find the container with id f28ee883dbe1fd6ba5aefd758ce04fe5d70f514e06714685b9af71ccc86d8630 Oct 31 23:13:40.612734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.612709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:40.719793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.718334 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq"] Oct 31 23:13:40.726893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.726021 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.760341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.760258 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq"] Oct 31 23:13:40.826974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.826936 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.827201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.827176 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.827436 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.827415 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzjn\" (UniqueName: \"kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.929671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.929629 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.929835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.929803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzjn\" (UniqueName: \"kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.929914 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.929874 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.930749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.930714 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.932762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.932736 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:40.943024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:40.942997 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzjn\" (UniqueName: \"kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn\") pod \"maas-api-revoke-sub-91cff7ca-kwzzq\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:41.054203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.053751 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:41.298619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.298513 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" event={"ID":"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a","Type":"ContainerStarted","Data":"ec916a0a1c9298d580dca1b7bc46bade1aa23c823e05ade58376020fa021aeae"} Oct 31 23:13:41.298619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.298613 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" event={"ID":"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a","Type":"ContainerStarted","Data":"f28ee883dbe1fd6ba5aefd758ce04fe5d70f514e06714685b9af71ccc86d8630"} Oct 31 23:13:41.301830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.301795 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" event={"ID":"b2d8b699-311d-48f7-be78-2223ba357359","Type":"ContainerStarted","Data":"dc71b80747ace761be5e652bcbcb219230b36bc09e066e6b83a1ab2c150d4b93"} Oct 31 23:13:41.301955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.301851 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" event={"ID":"b2d8b699-311d-48f7-be78-2223ba357359","Type":"ContainerStarted","Data":"8d36526d60e14a955eb138e33b129060d2f8f0bd160c70196d5136e7d112e71e"} Oct 31 23:13:41.304815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.304739 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" event={"ID":"3a9dcb05-0ae4-4b58-bed7-39549e5800cd","Type":"ContainerStarted","Data":"75bda808064c36c6b90b000fde31553ae9897010a81d8e284434457a9f1d6237"} Oct 31 23:13:41.304815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.304785 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" event={"ID":"3a9dcb05-0ae4-4b58-bed7-39549e5800cd","Type":"ContainerStarted","Data":"0fbe55994721041c7a6ebdfa67a2f6cccc00d89d08eefb4fb1153185c740b2dc"} Oct 31 23:13:41.360730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.360664 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" podStartSLOduration=2.360645427 podStartE2EDuration="2.360645427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:41.323076229 +0000 UTC m=+1447.552630642" watchObservedRunningTime="2026-10-31 23:13:41.360645427 +0000 UTC m=+1447.590199839" Oct 31 23:13:41.371209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.371157 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podStartSLOduration=2.371138911 podStartE2EDuration="2.371138911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:41.357131862 +0000 UTC m=+1447.586686275" watchObservedRunningTime="2026-10-31 23:13:41.371138911 +0000 UTC m=+1447.600693323" Oct 31 23:13:41.377496 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:41.377476 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df1b859_5913_4192_8be1_5a8577bdc554.slice/crio-582b1f87b073ed8ec0e064d7911ac2091aff574dcc5828abd62b596b206106ef WatchSource:0}: Error finding container 582b1f87b073ed8ec0e064d7911ac2091aff574dcc5828abd62b596b206106ef: Status 404 returned error can't find the container with id 582b1f87b073ed8ec0e064d7911ac2091aff574dcc5828abd62b596b206106ef Oct 31 23:13:41.377922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.377901 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq"] Oct 31 23:13:41.381477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.381443 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podStartSLOduration=2.381430369 podStartE2EDuration="2.381430369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:41.379737875 +0000 UTC m=+1447.609292287" watchObservedRunningTime="2026-10-31 23:13:41.381430369 +0000 UTC m=+1447.610984780" Oct 31 23:13:41.501889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.501848 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:41.983168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.982097 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:41.983168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:41.982178 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:42.002388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.002360 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:42.111724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.111672 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:42.112652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.112623 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:42.112784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.112674 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:42.245571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.243258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:42.472569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.472232 5043 generic.go:356] "Generic (PLEG): container finished" podID="d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" containerID="ec916a0a1c9298d580dca1b7bc46bade1aa23c823e05ade58376020fa021aeae" exitCode=0 Oct 31 23:13:42.483576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.479394 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" event={"ID":"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a","Type":"ContainerDied","Data":"ec916a0a1c9298d580dca1b7bc46bade1aa23c823e05ade58376020fa021aeae"} Oct 31 23:13:42.490880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.490836 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" event={"ID":"7df1b859-5913-4192-8be1-5a8577bdc554","Type":"ContainerStarted","Data":"57393695f034569221bf98f7c5704425eef6c01f009378a81b74b10e657d3c20"} Oct 31 23:13:42.491031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.490913 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" event={"ID":"7df1b859-5913-4192-8be1-5a8577bdc554","Type":"ContainerStarted","Data":"582b1f87b073ed8ec0e064d7911ac2091aff574dcc5828abd62b596b206106ef"} Oct 31 23:13:42.529450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.525254 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" podStartSLOduration=2.525237611 podStartE2EDuration="2.525237611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:42.5242281 +0000 UTC m=+1448.753782513" watchObservedRunningTime="2026-10-31 23:13:42.525237611 +0000 UTC m=+1448.754792024" Oct 31 23:13:42.700482 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:13:42.700451 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df1b859_5913_4192_8be1_5a8577bdc554.slice/crio-57393695f034569221bf98f7c5704425eef6c01f009378a81b74b10e657d3c20.scope\": RecentStats: unable to find data in memory cache]" Oct 31 23:13:42.982018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.981929 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:42.982253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:42.982228 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:43.007409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.007374 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w"] Oct 31 23:13:43.012385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.012367 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.045606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.040719 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:43.076567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.063075 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w"] Oct 31 23:13:43.119568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.106062 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt24x\" (UniqueName: \"kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.119568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.106256 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.119568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.106314 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.173525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.164783 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:43.173525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.164865 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:43.245615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.207705 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt24x\" (UniqueName: \"kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.245615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.207925 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.245615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.207992 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.245615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.213862 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.252567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.251449 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.305319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.305195 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt24x\" (UniqueName: \"kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x\") pod \"maas-api-revoke-sub-5ad6e216-xrs5w\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.365569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.364773 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:43.535623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.513699 5043 generic.go:356] "Generic (PLEG): container finished" podID="7df1b859-5913-4192-8be1-5a8577bdc554" containerID="57393695f034569221bf98f7c5704425eef6c01f009378a81b74b10e657d3c20" exitCode=0 Oct 31 23:13:43.535623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:43.515369 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" event={"ID":"7df1b859-5913-4192-8be1-5a8577bdc554","Type":"ContainerDied","Data":"57393695f034569221bf98f7c5704425eef6c01f009378a81b74b10e657d3c20"} Oct 31 23:13:44.091661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.061874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:44.108387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.106766 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:44.108387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.106821 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:44.108841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.108808 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w"] Oct 31 23:13:44.165637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.139110 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:44.166200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.166157 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:44.496589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.495628 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:44.604361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.604294 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" Oct 31 23:13:44.604987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.604306 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk" event={"ID":"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a","Type":"ContainerDied","Data":"f28ee883dbe1fd6ba5aefd758ce04fe5d70f514e06714685b9af71ccc86d8630"} Oct 31 23:13:44.604987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.604467 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f28ee883dbe1fd6ba5aefd758ce04fe5d70f514e06714685b9af71ccc86d8630" Oct 31 23:13:44.606453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.606435 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw\" (UniqueName: \"kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw\") pod \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " Oct 31 23:13:44.607026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.606881 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" event={"ID":"ac3f856c-94b7-4519-8b01-1b67fb76afb1","Type":"ContainerStarted","Data":"2fe0405dff0fe3ab6e92d8e9fb8548cad253e0582033e222c01f8ebb65e6aa98"} Oct 31 23:13:44.607273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.607253 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca\") pod \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " Oct 31 23:13:44.607948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.607536 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca" pod "d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" (UID: "d0320a6d-9fd8-4704-b0f7-5db4bfc6307a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:44.607948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.607798 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca\") pod \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\" (UID: \"d0320a6d-9fd8-4704-b0f7-5db4bfc6307a\") " Oct 31 23:13:44.608468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.608448 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:44.634359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.634275 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca" pod "d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" (UID: "d0320a6d-9fd8-4704-b0f7-5db4bfc6307a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:44.661228 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.643101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:44.770612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.761532 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw" pod "d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" (UID: "d0320a6d-9fd8-4704-b0f7-5db4bfc6307a"). InnerVolumeSpecName "kube-api-access-tzjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:44.802063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.797648 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjzw\" (UniqueName: \"kubernetes.io/projected/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-kube-api-access-tzjzw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:44.802063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.797716 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:44.993502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.993399 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:44.993851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:44.993788 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:45.112571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.112513 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:45.112812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.112777 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:45.193843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.193784 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:45.207760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.207503 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca\") pod \"7df1b859-5913-4192-8be1-5a8577bdc554\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " Oct 31 23:13:45.207760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.207590 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn\" (UniqueName: \"kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn\") pod \"7df1b859-5913-4192-8be1-5a8577bdc554\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " Oct 31 23:13:45.207963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.207825 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca\") pod \"7df1b859-5913-4192-8be1-5a8577bdc554\" (UID: \"7df1b859-5913-4192-8be1-5a8577bdc554\") " Oct 31 23:13:45.207963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.207893 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca" pod "7df1b859-5913-4192-8be1-5a8577bdc554" (UID: "7df1b859-5913-4192-8be1-5a8577bdc554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:45.208618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.208598 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:45.209877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.209850 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn" pod "7df1b859-5913-4192-8be1-5a8577bdc554" (UID: "7df1b859-5913-4192-8be1-5a8577bdc554"). InnerVolumeSpecName "kube-api-access-bqzjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:45.209984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.209964 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca" pod "7df1b859-5913-4192-8be1-5a8577bdc554" (UID: "7df1b859-5913-4192-8be1-5a8577bdc554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:45.310181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.310145 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzjn\" (UniqueName: \"kubernetes.io/projected/7df1b859-5913-4192-8be1-5a8577bdc554-kube-api-access-bqzjn\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:45.310181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.310176 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df1b859-5913-4192-8be1-5a8577bdc554-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:45.337327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.337293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:45.683241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.683195 5043 generic.go:356] "Generic (PLEG): container finished" podID="ac3f856c-94b7-4519-8b01-1b67fb76afb1" containerID="2db53418ce48d5ec216a92c23ddad4a0fce7ce90e7f78a29a260a4254e8f809d" exitCode=0 Oct 31 23:13:45.684821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.683474 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" event={"ID":"ac3f856c-94b7-4519-8b01-1b67fb76afb1","Type":"ContainerDied","Data":"2db53418ce48d5ec216a92c23ddad4a0fce7ce90e7f78a29a260a4254e8f809d"} Oct 31 23:13:45.687518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.687228 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" event={"ID":"7df1b859-5913-4192-8be1-5a8577bdc554","Type":"ContainerDied","Data":"582b1f87b073ed8ec0e064d7911ac2091aff574dcc5828abd62b596b206106ef"} Oct 31 23:13:45.687518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.687287 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="582b1f87b073ed8ec0e064d7911ac2091aff574dcc5828abd62b596b206106ef" Oct 31 23:13:45.687518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.687235 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq" Oct 31 23:13:45.978841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.978590 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:45.978841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:45.978681 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:46.113588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:46.112049 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:46.113588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:46.112107 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:46.137393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:46.137356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:46.982627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:46.977949 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:46.982627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:46.978013 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:47.055569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.053145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:47.124903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.124375 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:47.124903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.124441 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:47.806272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.746846 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:47.972673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.946025 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:47.972673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.946618 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:47.973819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.973408 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.001623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.978607 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:48.001623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:47.978664 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:48.086017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.085918 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.273587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.269191 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:48.273587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.269260 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:48.274258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.274217 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:13:48.283726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.283687 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.471625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.457615 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.488037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.487994 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.516996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.510032 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:13:48.517394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.517363 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:48.600578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.599195 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.620652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.611644 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.748630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.747754 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:48.765566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.753084 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:13:48.792922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.782339 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca\") pod \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " Oct 31 23:13:48.792922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.782464 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x\" (UniqueName: \"kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x\") pod \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " Oct 31 23:13:48.792922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.782605 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca\") pod \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\" (UID: \"ac3f856c-94b7-4519-8b01-1b67fb76afb1\") " Oct 31 23:13:48.911892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.905934 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca" pod "ac3f856c-94b7-4519-8b01-1b67fb76afb1" (UID: "ac3f856c-94b7-4519-8b01-1b67fb76afb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:48.911892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.911728 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x" pod "ac3f856c-94b7-4519-8b01-1b67fb76afb1" (UID: "ac3f856c-94b7-4519-8b01-1b67fb76afb1"). InnerVolumeSpecName "kube-api-access-zt24x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:49.035907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.953075 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:49.054716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.953145 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:49.055578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:48.985874 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.056395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.018683 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt24x\" (UniqueName: \"kubernetes.io/projected/ac3f856c-94b7-4519-8b01-1b67fb76afb1-kube-api-access-zt24x\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:49.071741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.071702 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:49.071926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.068900 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:13:49.072175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.072158 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:13:49.088623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.088014 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.088623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.088312 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:13:49.089516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.089483 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.090065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.090040 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:13:49.146584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.141007 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca" pod "ac3f856c-94b7-4519-8b01-1b67fb76afb1" (UID: "ac3f856c-94b7-4519-8b01-1b67fb76afb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:13:49.146584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.142304 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:49.146584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.142354 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:49.184602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.151651 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:49.184602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.176618 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:49.184602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.178184 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac3f856c-94b7-4519-8b01-1b67fb76afb1-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:49.317749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.317632 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.358669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.358608 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.399581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.398113 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:13:49.403968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.403852 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:13:49.590485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.572634 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.590485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.573207 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:13:49.590485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.574625 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.590485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.575126 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:13:49.602393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.601115 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.602393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.601967 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:13:49.624789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.624745 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:49.642995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.642961 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:13:49.805570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:49.804643 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:50.027979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.020671 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 31 23:13:50.040944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.040678 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:50.040944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.040742 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:50.087651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.061451 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.088671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.088637 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:13:50.096639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.096608 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:50.186423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.186379 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.204654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.193934 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.391103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.391003 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" Oct 31 23:13:50.452597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.417699 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.452597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.448936 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:13:50.452597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.448977 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w" event={"ID":"ac3f856c-94b7-4519-8b01-1b67fb76afb1","Type":"ContainerDied","Data":"2fe0405dff0fe3ab6e92d8e9fb8548cad253e0582033e222c01f8ebb65e6aa98"} Oct 31 23:13:50.452597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.449017 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe0405dff0fe3ab6e92d8e9fb8548cad253e0582033e222c01f8ebb65e6aa98" Oct 31 23:13:50.458568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.455357 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:50.458842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.458813 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:50.484572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.459944 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.601138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.601090 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.602757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.602724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:50.605127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.605096 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:50.675041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.674944 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:50.829196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:50.829139 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:51.185698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.185647 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.218750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.216475 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:13:51.218750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.216821 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:13:51.297586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.229258 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.301568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.301173 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.305568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.302119 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:51.352668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.346806 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 23:13:51.352668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.346866 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 23:13:51.508293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.480712 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.508293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.486461 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.508293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.507686 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.524700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.524663 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:51.524951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.524922 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:51.581732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.581682 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.689605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.667686 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:13:51.703664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.669248 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:51.704918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.669313 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.709569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.706259 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:51.739093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.739060 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:51.739692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.739629 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:51.832837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:51.832736 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.213624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.167815 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:52.213624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.167876 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:52.334573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.218789 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.225276 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.225318 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.295104 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.554012 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.554056 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.600844 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.637632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.554666 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.721989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.698319 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:13:52.722749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.722707 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:52.730233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.730192 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:52.752202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.745715 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.752202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.746098 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 23:13:52.752534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.752508 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:52.752715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.752673 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:52.844786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.844734 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.001581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:52.992125 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.130805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.130759 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 23:13:53.131101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.131052 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 23:13:53.325773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.325671 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:53.351570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.329804 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.672631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.672469 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.697572 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:13:53.694026 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3f856c_94b7_4519_8b01_1b67fb76afb1.slice/crio-2fe0405dff0fe3ab6e92d8e9fb8548cad253e0582033e222c01f8ebb65e6aa98\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3f856c_94b7_4519_8b01_1b67fb76afb1.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:13:53.717721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.717668 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:13:53.782654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.766621 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.782654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.779785 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.782654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.780143 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:13:53.785787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.783860 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.785787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.784163 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:53.785787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.785131 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:13:53.804591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.786349 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.804591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.786627 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:53.804591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.786881 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:53.876786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.860779 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:53.907737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.907691 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:53.907957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:53.907761 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.111284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.111181 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.488625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.487956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:54.544369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.534962 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:54.544369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.535031 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.703575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.635734 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:13:54.703575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.684453 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.756258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.733121 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:13:54.775798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.775755 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.814211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.814163 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:13:54.815702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.815666 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:54.816888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.816863 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:13:54.817272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.817250 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:54.818334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.818310 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:13:54.823793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.819324 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:54.867972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.843687 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 31 23:13:54.869598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.868227 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:54.895595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.873037 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.895595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.873611 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:54.963435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.959788 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:54.963435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.959868 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:54.983059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.983018 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:54.983335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:54.983300 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.079628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.054749 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:55.163676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.140781 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:55.225572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.221727 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:55.361347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.359641 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:55.361347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.359707 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.486119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.486075 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.543739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.543702 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:55.544319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.544286 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.587027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.586997 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq"] Oct 31 23:13:55.587886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.587856 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:55.588095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.588072 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.625350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.625262 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ac3f856c-94b7-4519-8b01-1b67fb76afb1" containerName="revoke-keys" Oct 31 23:13:55.655700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.655673 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3f856c-94b7-4519-8b01-1b67fb76afb1" containerName="revoke-keys" Oct 31 23:13:55.662932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.662883 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7df1b859-5913-4192-8be1-5a8577bdc554" containerName="revoke-keys" Oct 31 23:13:55.663615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.663593 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df1b859-5913-4192-8be1-5a8577bdc554" containerName="revoke-keys" Oct 31 23:13:55.663891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.663877 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" containerName="revoke-keys" Oct 31 23:13:55.664113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.664099 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" containerName="revoke-keys" Oct 31 23:13:55.664488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.664475 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3f856c-94b7-4519-8b01-1b67fb76afb1" containerName="revoke-keys" Oct 31 23:13:55.667818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.667800 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df1b859-5913-4192-8be1-5a8577bdc554" containerName="revoke-keys" Oct 31 23:13:55.667947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.667937 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" containerName="revoke-keys" Oct 31 23:13:55.672946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.672922 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.696644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.687351 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq"] Oct 31 23:13:55.696644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.687714 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:13:55.704577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.703040 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:13:55.704577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.703144 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:13:55.705231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.704953 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:13:55.705231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.705043 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:13:55.763649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.749876 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.817261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.780225 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.819329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.819291 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.854242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.854193 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnlnf\" (UniqueName: \"kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.854525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.854508 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.854897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.854877 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.855206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.855190 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq"] Oct 31 23:13:55.869096 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.869057 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.869597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.869570 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:13:55.870012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.869988 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:55.873652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.873625 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:13:55.880686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.880614 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.901762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.895566 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.901762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.896735 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.901762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.898532 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:13:55.940274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.937637 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:55.980847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.980798 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnlnf\" (UniqueName: \"kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.981133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.981109 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.997914 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.997858 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:55.999284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:55.999253 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:56.015735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.015697 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:56.102815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.102782 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq"] Oct 31 23:13:56.147651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.104035 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.160581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.159280 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:56.189572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.187364 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnlnf\" (UniqueName: \"kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf\") pod \"maas-api-revoke-sub-27ecdf06-zhpzq\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:56.244776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.244739 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndnx\" (UniqueName: \"kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.245234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.245211 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.245614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.245594 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.421617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.373521 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.424800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.424756 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndnx\" (UniqueName: \"kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.445134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.443668 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:13:56.445134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.443746 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:56.477571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.467339 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndnx\" (UniqueName: \"kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.477571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.475747 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.477571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.477298 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.477571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.385944 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8da9f27-kx6jq\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.662484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.662442 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:56.669816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.669788 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:13:56.698845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.698760 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:56.699007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.698877 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:13:56.746636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.746602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:56.849740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.847840 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:56.894718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.894680 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:13:56.904360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.903715 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:13:56.904360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.904114 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:13:56.927034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.926987 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:56.930691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.930646 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:56.988139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.977926 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:13:56.988139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.933335 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:56.988139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.953902 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:13:56.988139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.972640 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:13:56.988139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.972737 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:13:56.988139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:56.980060 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:57.006132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.006036 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:13:57.029609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.026337 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:13:57.029609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.026639 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:13:57.034296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.034089 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:13:57.067717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.063047 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:13:57.087831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.087686 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:13:57.113819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.113792 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:13:57.113926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.113880 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:13:57.148928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.148901 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:13:57.213162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.212988 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:13:57.214443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.214379 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:13:57.301160 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:57.301094 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8998023d_f6f6_4afa_aa08_c09d4d371ff4.slice/crio-2598521798eb59c4fe2dcd52d6edf8e35d661a858d78747bdda52af269d5a1b3 WatchSource:0}: Error finding container 2598521798eb59c4fe2dcd52d6edf8e35d661a858d78747bdda52af269d5a1b3: Status 404 returned error can't find the container with id 2598521798eb59c4fe2dcd52d6edf8e35d661a858d78747bdda52af269d5a1b3 Oct 31 23:13:57.315336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.314957 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq"] Oct 31 23:13:57.428016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.427979 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz"] Oct 31 23:13:57.450430 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:57.449865 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aae16c_c087_4ef8_8d50_da1bcaab927a.slice/crio-26f64df14168cccf9e7e466df0a4e8013d01d1181d9ae8ca6115e8721dfc4071 WatchSource:0}: Error finding container 26f64df14168cccf9e7e466df0a4e8013d01d1181d9ae8ca6115e8721dfc4071: Status 404 returned error can't find the container with id 26f64df14168cccf9e7e466df0a4e8013d01d1181d9ae8ca6115e8721dfc4071 Oct 31 23:13:57.455745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.455275 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.459015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.458954 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz"] Oct 31 23:13:57.472380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.472192 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq"] Oct 31 23:13:57.479497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.479439 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" event={"ID":"8998023d-f6f6-4afa-aa08-c09d4d371ff4","Type":"ContainerStarted","Data":"2598521798eb59c4fe2dcd52d6edf8e35d661a858d78747bdda52af269d5a1b3"} Oct 31 23:13:57.487611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.487584 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:57.553892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.553856 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2px\" (UniqueName: \"kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.554063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.553953 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.554719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.554066 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.656176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.656139 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.656504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.656480 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2px\" (UniqueName: \"kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.656620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.656588 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.657192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.657172 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.658416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.658391 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.669197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.669167 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2px\" (UniqueName: \"kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px\") pod \"maas-api-revoke-sub-3bcdb010-mpvjz\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:57.783870 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:57.783827 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:13:58.087449 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:13:58.087416 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c1b48b_4e73_4d61_bd7a_36689a3a06fe.slice/crio-616131a3ecca9f5d2a205403f2c6ac6dfabaea50b992fa51774a5b02cc2dafc1 WatchSource:0}: Error finding container 616131a3ecca9f5d2a205403f2c6ac6dfabaea50b992fa51774a5b02cc2dafc1: Status 404 returned error can't find the container with id 616131a3ecca9f5d2a205403f2c6ac6dfabaea50b992fa51774a5b02cc2dafc1 Oct 31 23:13:58.089009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.088946 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz"] Oct 31 23:13:58.238198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.238164 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:58.499999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.499969 5043 generic.go:356] "Generic (PLEG): container finished" podID="8998023d-f6f6-4afa-aa08-c09d4d371ff4" containerID="3c8cb29237b873b0eebc56508749bc30b4b774b2bbae682727533f504f0ecb9d" exitCode=0 Oct 31 23:13:58.500185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.500053 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" event={"ID":"8998023d-f6f6-4afa-aa08-c09d4d371ff4","Type":"ContainerDied","Data":"3c8cb29237b873b0eebc56508749bc30b4b774b2bbae682727533f504f0ecb9d"} Oct 31 23:13:58.501552 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.501523 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" event={"ID":"60c1b48b-4e73-4d61-bd7a-36689a3a06fe","Type":"ContainerStarted","Data":"616131a3ecca9f5d2a205403f2c6ac6dfabaea50b992fa51774a5b02cc2dafc1"} Oct 31 23:13:58.503259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.503233 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" event={"ID":"33aae16c-c087-4ef8-8d50-da1bcaab927a","Type":"ContainerStarted","Data":"fd4e4199fe4e90633e8994d445ddb7fabfc0e18bdd26845ce6a61e48569db1e7"} Oct 31 23:13:58.503358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.503282 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" event={"ID":"33aae16c-c087-4ef8-8d50-da1bcaab927a","Type":"ContainerStarted","Data":"26f64df14168cccf9e7e466df0a4e8013d01d1181d9ae8ca6115e8721dfc4071"} Oct 31 23:13:58.545226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.545170 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" podStartSLOduration=7.545152 podStartE2EDuration="7.545152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:13:58.539473926 +0000 UTC m=+1464.769028338" watchObservedRunningTime="2026-10-31 23:13:58.545152 +0000 UTC m=+1464.774706412" Oct 31 23:13:58.948119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:58.948081 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:59.515534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.515499 5043 generic.go:356] "Generic (PLEG): container finished" podID="60c1b48b-4e73-4d61-bd7a-36689a3a06fe" containerID="dd30efa8a12e0610cfc599f98ec5b9f0b7afd478359b89c0aacbb2e6e2ee7306" exitCode=0 Oct 31 23:13:59.515766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.515616 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" event={"ID":"60c1b48b-4e73-4d61-bd7a-36689a3a06fe","Type":"ContainerDied","Data":"dd30efa8a12e0610cfc599f98ec5b9f0b7afd478359b89c0aacbb2e6e2ee7306"} Oct 31 23:13:59.517501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.517455 5043 generic.go:356] "Generic (PLEG): container finished" podID="33aae16c-c087-4ef8-8d50-da1bcaab927a" containerID="fd4e4199fe4e90633e8994d445ddb7fabfc0e18bdd26845ce6a61e48569db1e7" exitCode=0 Oct 31 23:13:59.517660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.517490 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" event={"ID":"33aae16c-c087-4ef8-8d50-da1bcaab927a","Type":"ContainerDied","Data":"fd4e4199fe4e90633e8994d445ddb7fabfc0e18bdd26845ce6a61e48569db1e7"} Oct 31 23:13:59.677254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.677204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:13:59.797660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.797635 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:13:59.926842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.926804 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx\" (UniqueName: \"kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx\") pod \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " Oct 31 23:13:59.927101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.927084 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca\") pod \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " Oct 31 23:13:59.927181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.927146 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca\") pod \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\" (UID: \"8998023d-f6f6-4afa-aa08-c09d4d371ff4\") " Oct 31 23:13:59.927469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.927423 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca" pod "8998023d-f6f6-4afa-aa08-c09d4d371ff4" (UID: "8998023d-f6f6-4afa-aa08-c09d4d371ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:13:59.927996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.927967 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:13:59.929522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.929488 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx" pod "8998023d-f6f6-4afa-aa08-c09d4d371ff4" (UID: "8998023d-f6f6-4afa-aa08-c09d4d371ff4"). InnerVolumeSpecName "kube-api-access-sndnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:13:59.929522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:13:59.929512 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca" pod "8998023d-f6f6-4afa-aa08-c09d4d371ff4" (UID: "8998023d-f6f6-4afa-aa08-c09d4d371ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:00.029894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.029810 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sndnx\" (UniqueName: \"kubernetes.io/projected/8998023d-f6f6-4afa-aa08-c09d4d371ff4-kube-api-access-sndnx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.029894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.029841 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8998023d-f6f6-4afa-aa08-c09d4d371ff4-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.340334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.340229 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:00.402330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.402288 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh"] Oct 31 23:14:00.403259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.403235 5043 cpu_manager.go:414] "containerMap: removing container" podUID="8998023d-f6f6-4afa-aa08-c09d4d371ff4" containerName="revoke-keys" Oct 31 23:14:00.403387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.403260 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8998023d-f6f6-4afa-aa08-c09d4d371ff4" containerName="revoke-keys" Oct 31 23:14:00.403501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.403488 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8998023d-f6f6-4afa-aa08-c09d4d371ff4" containerName="revoke-keys" Oct 31 23:14:00.407352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.407323 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.437608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.428530 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh"] Oct 31 23:14:00.529334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.529300 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" Oct 31 23:14:00.529488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.529335 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq" event={"ID":"8998023d-f6f6-4afa-aa08-c09d4d371ff4","Type":"ContainerDied","Data":"2598521798eb59c4fe2dcd52d6edf8e35d661a858d78747bdda52af269d5a1b3"} Oct 31 23:14:00.529488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.529385 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2598521798eb59c4fe2dcd52d6edf8e35d661a858d78747bdda52af269d5a1b3" Oct 31 23:14:00.547223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.547182 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.547406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.547351 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.547406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.547390 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wshfr\" (UniqueName: \"kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.659594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.649337 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wshfr\" (UniqueName: \"kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.659594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.649716 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.659594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.649827 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.659594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.651166 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.676671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.668992 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wshfr\" (UniqueName: \"kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.678165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.678131 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86fa2ff4-m8lnh\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.745701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.745670 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:00.776742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.776721 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:14:00.779526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.779504 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:14:00.869635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.863131 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca\") pod \"33aae16c-c087-4ef8-8d50-da1bcaab927a\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " Oct 31 23:14:00.869635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.863275 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca\") pod \"33aae16c-c087-4ef8-8d50-da1bcaab927a\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " Oct 31 23:14:00.869635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.863348 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px\" (UniqueName: \"kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px\") pod \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " Oct 31 23:14:00.869635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.863396 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca\") pod \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " Oct 31 23:14:00.869635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.863691 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca\") pod \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\" (UID: \"60c1b48b-4e73-4d61-bd7a-36689a3a06fe\") " Oct 31 23:14:00.869635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.863766 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf\" (UniqueName: \"kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf\") pod \"33aae16c-c087-4ef8-8d50-da1bcaab927a\" (UID: \"33aae16c-c087-4ef8-8d50-da1bcaab927a\") " Oct 31 23:14:00.873638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.873217 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca" pod "33aae16c-c087-4ef8-8d50-da1bcaab927a" (UID: "33aae16c-c087-4ef8-8d50-da1bcaab927a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:00.874182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.874158 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca" pod "60c1b48b-4e73-4d61-bd7a-36689a3a06fe" (UID: "60c1b48b-4e73-4d61-bd7a-36689a3a06fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:00.876204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.876182 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px" pod "60c1b48b-4e73-4d61-bd7a-36689a3a06fe" (UID: "60c1b48b-4e73-4d61-bd7a-36689a3a06fe"). InnerVolumeSpecName "kube-api-access-8v2px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:00.877473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.877449 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf" pod "33aae16c-c087-4ef8-8d50-da1bcaab927a" (UID: "33aae16c-c087-4ef8-8d50-da1bcaab927a"). InnerVolumeSpecName "kube-api-access-bnlnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:00.877833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.877811 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca" pod "60c1b48b-4e73-4d61-bd7a-36689a3a06fe" (UID: "60c1b48b-4e73-4d61-bd7a-36689a3a06fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:00.879924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.879895 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca" pod "33aae16c-c087-4ef8-8d50-da1bcaab927a" (UID: "33aae16c-c087-4ef8-8d50-da1bcaab927a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:00.966373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.966330 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.966373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.966366 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnlnf\" (UniqueName: \"kubernetes.io/projected/33aae16c-c087-4ef8-8d50-da1bcaab927a-kube-api-access-bnlnf\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.966373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.966377 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.966633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.966386 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33aae16c-c087-4ef8-8d50-da1bcaab927a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.966633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.966396 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2px\" (UniqueName: \"kubernetes.io/projected/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-kube-api-access-8v2px\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:00.966633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:00.966404 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c1b48b-4e73-4d61-bd7a-36689a3a06fe-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:01.067859 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:01.067821 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a714fb_e0bb_45e8_a5d2_5ea07beb583f.slice/crio-ef30c76f2ea4907a086fff726981c6f76915304ba8c646ce8c470fddd0fa32bd WatchSource:0}: Error finding container ef30c76f2ea4907a086fff726981c6f76915304ba8c646ce8c470fddd0fa32bd: Status 404 returned error can't find the container with id ef30c76f2ea4907a086fff726981c6f76915304ba8c646ce8c470fddd0fa32bd Oct 31 23:14:01.069359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.069331 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh"] Oct 31 23:14:01.116409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.116357 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:01.541850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.541810 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" event={"ID":"60c1b48b-4e73-4d61-bd7a-36689a3a06fe","Type":"ContainerDied","Data":"616131a3ecca9f5d2a205403f2c6ac6dfabaea50b992fa51774a5b02cc2dafc1"} Oct 31 23:14:01.541850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.541840 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz" Oct 31 23:14:01.542081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.541872 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="616131a3ecca9f5d2a205403f2c6ac6dfabaea50b992fa51774a5b02cc2dafc1" Oct 31 23:14:01.543700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.543677 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" Oct 31 23:14:01.543700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.543686 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq" event={"ID":"33aae16c-c087-4ef8-8d50-da1bcaab927a","Type":"ContainerDied","Data":"26f64df14168cccf9e7e466df0a4e8013d01d1181d9ae8ca6115e8721dfc4071"} Oct 31 23:14:01.543906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.543731 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f64df14168cccf9e7e466df0a4e8013d01d1181d9ae8ca6115e8721dfc4071" Oct 31 23:14:01.545797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.545773 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" event={"ID":"37a714fb-e0bb-45e8-a5d2-5ea07beb583f","Type":"ContainerStarted","Data":"35028b5c725596f0c7eb5316b6784342542a9b695b9c71bac261142a135a2c33"} Oct 31 23:14:01.545974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.545953 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" event={"ID":"37a714fb-e0bb-45e8-a5d2-5ea07beb583f","Type":"ContainerStarted","Data":"ef30c76f2ea4907a086fff726981c6f76915304ba8c646ce8c470fddd0fa32bd"} Oct 31 23:14:01.567502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.567456 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" podStartSLOduration=1.567441232 podStartE2EDuration="1.567441232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:01.566763671 +0000 UTC m=+1467.796318084" watchObservedRunningTime="2026-10-31 23:14:01.567441232 +0000 UTC m=+1467.796995642" Oct 31 23:14:01.733250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:01.718831 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:02.031443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.031409 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5"] Oct 31 23:14:02.033018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.032987 5043 cpu_manager.go:414] "containerMap: removing container" podUID="33aae16c-c087-4ef8-8d50-da1bcaab927a" containerName="revoke-keys" Oct 31 23:14:02.033018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.033015 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33aae16c-c087-4ef8-8d50-da1bcaab927a" containerName="revoke-keys" Oct 31 23:14:02.033291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.033053 5043 cpu_manager.go:414] "containerMap: removing container" podUID="60c1b48b-4e73-4d61-bd7a-36689a3a06fe" containerName="revoke-keys" Oct 31 23:14:02.033291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.033062 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c1b48b-4e73-4d61-bd7a-36689a3a06fe" containerName="revoke-keys" Oct 31 23:14:02.033461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.033299 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c1b48b-4e73-4d61-bd7a-36689a3a06fe" containerName="revoke-keys" Oct 31 23:14:02.033461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.033342 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="33aae16c-c087-4ef8-8d50-da1bcaab927a" containerName="revoke-keys" Oct 31 23:14:02.039721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.039689 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.041972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.041944 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5"] Oct 31 23:14:02.088715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.088678 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.089173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.088981 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.089173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.089066 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2k2\" (UniqueName: \"kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.190528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.190495 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2k2\" (UniqueName: \"kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.190716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.190658 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.190843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.190828 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.191431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.191403 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.193130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.193109 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.203404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.203382 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2k2\" (UniqueName: \"kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2\") pod \"maas-api-revoke-sub-09922503-rc6s5\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.361703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.361626 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:02.495353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.495043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:02.588268 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.588235 5043 generic.go:356] "Generic (PLEG): container finished" podID="37a714fb-e0bb-45e8-a5d2-5ea07beb583f" containerID="35028b5c725596f0c7eb5316b6784342542a9b695b9c71bac261142a135a2c33" exitCode=0 Oct 31 23:14:02.588435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.588415 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" event={"ID":"37a714fb-e0bb-45e8-a5d2-5ea07beb583f","Type":"ContainerDied","Data":"35028b5c725596f0c7eb5316b6784342542a9b695b9c71bac261142a135a2c33"} Oct 31 23:14:02.663969 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:02.663935 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf858a4d_b7eb_4d98_bba4_8e29f15b6791.slice/crio-532ab69f35ae5f69a979fde676a6e6cac2dbad4332143ca6e570b8bd736a45e7 WatchSource:0}: Error finding container 532ab69f35ae5f69a979fde676a6e6cac2dbad4332143ca6e570b8bd736a45e7: Status 404 returned error can't find the container with id 532ab69f35ae5f69a979fde676a6e6cac2dbad4332143ca6e570b8bd736a45e7 Oct 31 23:14:02.664897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:02.664872 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5"] Oct 31 23:14:03.136399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.135639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:03.600504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.600452 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" event={"ID":"af858a4d-b7eb-4d98-bba4-8e29f15b6791","Type":"ContainerStarted","Data":"733d30cfc50d6e923776884427989cfcfa602a668bd2d2474fad961001f43169"} Oct 31 23:14:03.600729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.600523 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" event={"ID":"af858a4d-b7eb-4d98-bba4-8e29f15b6791","Type":"ContainerStarted","Data":"532ab69f35ae5f69a979fde676a6e6cac2dbad4332143ca6e570b8bd736a45e7"} Oct 31 23:14:03.628791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.628729 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" podStartSLOduration=1.628709589 podStartE2EDuration="1.628709589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:03.62405378 +0000 UTC m=+1469.853608252" watchObservedRunningTime="2026-10-31 23:14:03.628709589 +0000 UTC m=+1469.858264001" Oct 31 23:14:03.786670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.786643 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:03.825073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.825033 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca\") pod \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " Oct 31 23:14:03.825248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.825123 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca\") pod \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " Oct 31 23:14:03.825348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.825329 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr\" (UniqueName: \"kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr\") pod \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\" (UID: \"37a714fb-e0bb-45e8-a5d2-5ea07beb583f\") " Oct 31 23:14:03.825430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.825404 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca" pod "37a714fb-e0bb-45e8-a5d2-5ea07beb583f" (UID: "37a714fb-e0bb-45e8-a5d2-5ea07beb583f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:03.826241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.826211 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:03.827398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.827370 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca" pod "37a714fb-e0bb-45e8-a5d2-5ea07beb583f" (UID: "37a714fb-e0bb-45e8-a5d2-5ea07beb583f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:03.828154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.828127 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr" pod "37a714fb-e0bb-45e8-a5d2-5ea07beb583f" (UID: "37a714fb-e0bb-45e8-a5d2-5ea07beb583f"). InnerVolumeSpecName "kube-api-access-wshfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:03.945630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.943865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:03.951400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.951073 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:03.951400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:03.951114 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wshfr\" (UniqueName: \"kubernetes.io/projected/37a714fb-e0bb-45e8-a5d2-5ea07beb583f-kube-api-access-wshfr\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:04.870073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:04.866928 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" event={"ID":"37a714fb-e0bb-45e8-a5d2-5ea07beb583f","Type":"ContainerDied","Data":"ef30c76f2ea4907a086fff726981c6f76915304ba8c646ce8c470fddd0fa32bd"} Oct 31 23:14:04.892578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:04.888818 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef30c76f2ea4907a086fff726981c6f76915304ba8c646ce8c470fddd0fa32bd" Oct 31 23:14:04.892578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:04.887804 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh" Oct 31 23:14:04.963582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:04.960933 5043 generic.go:356] "Generic (PLEG): container finished" podID="af858a4d-b7eb-4d98-bba4-8e29f15b6791" containerID="733d30cfc50d6e923776884427989cfcfa602a668bd2d2474fad961001f43169" exitCode=0 Oct 31 23:14:04.963582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:04.961296 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" event={"ID":"af858a4d-b7eb-4d98-bba4-8e29f15b6791","Type":"ContainerDied","Data":"733d30cfc50d6e923776884427989cfcfa602a668bd2d2474fad961001f43169"} Oct 31 23:14:05.612566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.612156 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:05.617570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.613958 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:05.651625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.650538 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:05.651625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.651239 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q"] Oct 31 23:14:05.653120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.652356 5043 cpu_manager.go:414] "containerMap: removing container" podUID="37a714fb-e0bb-45e8-a5d2-5ea07beb583f" containerName="revoke-keys" Oct 31 23:14:05.653120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.652378 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a714fb-e0bb-45e8-a5d2-5ea07beb583f" containerName="revoke-keys" Oct 31 23:14:05.659100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.656653 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a714fb-e0bb-45e8-a5d2-5ea07beb583f" containerName="revoke-keys" Oct 31 23:14:05.780597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.767893 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:05.800113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.792117 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:05.844603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.835238 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:05.844603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.836040 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:05.844603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.837065 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:05.856569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.853394 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:05.913625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.913425 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.039016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.992095 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdgs7\" (UniqueName: \"kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.039016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.992300 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.039016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:05.992370 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.334348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.102427 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.334348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.102518 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.334348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.102723 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdgs7\" (UniqueName: \"kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.334348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.268509 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.384093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.384050 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.384259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.384154 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:06.453926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.453872 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.501623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.490595 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.522572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.519277 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:06.565719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.565670 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.650647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.639694 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:14:06.682728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.640856 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:06.708935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.641694 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdgs7\" (UniqueName: \"kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7\") pod \"maas-api-revoke-sub-c02ccdc3-wr76q\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.709210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.676045 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.806571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.806484 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:06.829579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.826842 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.829579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.828731 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:06.838229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.838178 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.839656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.839618 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:06.844641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:06.844236 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:14:07.019662 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.078s" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.019773 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.019824 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.019863 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.019899 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.019920 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q"] Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.019979 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.020013 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.020045 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:14:07.021613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.020077 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:14:07.429569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.422135 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:07.429569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.422710 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:14:07.614157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.541760 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:07.614157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.542275 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:14:07.614157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.570958 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.614157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.571377 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:14:07.640568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.634332 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.692825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.684588 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.733759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.733708 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.737570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.734193 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:07.857573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.839317 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:14:07.869736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.866176 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:14:07.869736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.866134 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:07.869736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.867887 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:07.889836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.889784 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.918158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.918105 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:07.961916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:07.961868 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:08.532731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.532290 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:08.581306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.581256 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:08.597371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.597323 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:14:08.884850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.636170 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:08.884850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.882053 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:08.884850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.884321 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:08.900948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.900905 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:08.925724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.925672 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:14:08.942205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.942157 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:08.980571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.944619 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:08.981707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.944679 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:14:09.010571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.944701 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:09.081068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:08.993376 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.082796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.082759 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.083002 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.027291 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.586585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.546671 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.586585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.584415 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.588043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.587698 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.588043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.587755 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.724641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.719097 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.724907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.724874 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.888517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.884641 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.888517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.885021 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:09.888517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.885286 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:09.940696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.903711 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:09.940696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.926712 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:09.940696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.926775 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.940696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.927198 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:09.964571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.956017 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:09.982152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:09.982111 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.022590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.007627 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.048620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.025683 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.253615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.104708 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:10.332572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.105266 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:10.397324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.397285 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7"] Oct 31 23:14:10.561091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.561049 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.595575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.588595 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.615214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.614778 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:10.615392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.615360 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.765700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.765660 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:10.914261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.914156 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:10.997887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:10.997831 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.015013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.014951 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:11.016308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.016259 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:11.017449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.017415 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:11.029939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.029904 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:14:11.136940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.074792 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.137268 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.029848 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.137707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.110728 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:11.142774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.110849 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:11.161280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.142922 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.223486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.223392 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.223767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.223741 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.223944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.223923 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw9v4\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.224200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.224178 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.224337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.224320 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.224525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.224504 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.227590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.224773 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.227825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.227801 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.227962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.227945 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.400651 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.400736 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw9v4\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.400866 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.400895 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.400955 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.401065 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.401141 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.401180 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.418572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.401300 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.447571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.427498 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.447571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.427616 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:11.447571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.399431 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:11.472319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.472285 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.473461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.473436 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.553686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.543390 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.553947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.543674 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.555032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.554989 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:11.605888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.605827 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.677001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.652722 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:11.737099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.670179 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:11.737374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.737344 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.737651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.677857 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.737815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.677246 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.813805 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:14:11.813767 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:14:11.814028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.814012 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7"] Oct 31 23:14:11.859578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.845776 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.890446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.890401 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:11.937765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.937705 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:11.986573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:11.941533 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw9v4\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4\") pod \"e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:12.043510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.043472 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:12.044925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.044880 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:12.046181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.046139 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:12.138627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.115668 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.142775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.142735 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.179242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.161813 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.179587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.162288 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.316841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.214313 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:12.316841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.224190 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.316841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.266637 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:12.316841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.266707 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.404254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.404166 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5"] Oct 31 23:14:12.497807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.428362 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:12.498428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.463495 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.739682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.739632 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.792571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.765640 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:12.875629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.792839 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:12.875629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.872606 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:12.875629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:12.794222 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.008459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.008422 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:13.008794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.008745 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.046613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.043900 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:13.046613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.044087 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.046613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.045093 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:13.046613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.046040 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 31 23:14:13.046613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.046117 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:13.075333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.072834 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.142211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.135652 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.256900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.251081 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:13.304200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.304152 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:13.305375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.305341 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:14:13.315576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.307504 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:13.504570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.503864 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5"] Oct 31 23:14:13.512615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.509462 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:13.571746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.571697 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:13.662500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.648495 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.682774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.668264 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:13.683989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.683936 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:13.795642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.795128 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:14.168416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:13.926021 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:14.168416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.103849 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:14.168416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.106353 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:14.168416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.106538 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:14.198298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.172352 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:14.265569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.258290 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76mr8\" (UniqueName: \"kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:14.265939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.265905 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:14.269742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.269704 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:14.376069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.341808 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:14.398855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.386329 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:14:14.400040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.399961 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:14.413633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.401107 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:14.493572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.486715 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:14.527969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.527919 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded" Oct 31 23:14:14.561506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.561461 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:14.573467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.573421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:14.763256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.763215 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:14.774217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.774178 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:14.777045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.777011 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:14.928683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.928639 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 31 23:14:15.067618 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:15.063195 5043 logging.go:55] [core] [Channel #116 SubChannel #117]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:49586->10.132.0.59:50052: use of closed network connection" Oct 31 23:14:15.067618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.993014 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2\" (UniqueName: \"kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2\") pod \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " Oct 31 23:14:15.084698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:14.955825 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded" Oct 31 23:14:15.109590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.108691 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:15.249420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.151764 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:15.249420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.184759 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:15.249420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.196558 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:15.249420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.198966 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:15.273873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.273374 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2" pod "af858a4d-b7eb-4d98-bba4-8e29f15b6791" (UID: "af858a4d-b7eb-4d98-bba4-8e29f15b6791"). InnerVolumeSpecName "kube-api-access-cg2k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:15.276483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.275315 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca\") pod \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " Oct 31 23:14:15.276483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.276265 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76mr8\" (UniqueName: \"kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.276483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.276353 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.276483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.276453 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.277487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.277115 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2k2\" (UniqueName: \"kubernetes.io/projected/af858a4d-b7eb-4d98-bba4-8e29f15b6791-kube-api-access-cg2k2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:15.311582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.310763 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:15.314366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.313525 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.324643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.324580 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.328286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.328264 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76mr8\" (UniqueName: \"kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8\") pod \"maas-api-revoke-sub-472a38b0-gmdg5\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.330840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.314023 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca" pod "af858a4d-b7eb-4d98-bba4-8e29f15b6791" (UID: "af858a4d-b7eb-4d98-bba4-8e29f15b6791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:15.331994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.331715 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" Oct 31 23:14:15.370579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.370519 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:15.370716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.370625 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:15.370716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.370673 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:15.383074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.383047 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca\") pod \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\" (UID: \"af858a4d-b7eb-4d98-bba4-8e29f15b6791\") " Oct 31 23:14:15.384227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.384158 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:15.384717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.384695 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca" pod "af858a4d-b7eb-4d98-bba4-8e29f15b6791" (UID: "af858a4d-b7eb-4d98-bba4-8e29f15b6791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.389992 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5" event={"ID":"af858a4d-b7eb-4d98-bba4-8e29f15b6791","Type":"ContainerDied","Data":"532ab69f35ae5f69a979fde676a6e6cac2dbad4332143ca6e570b8bd736a45e7"} Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390051 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532ab69f35ae5f69a979fde676a6e6cac2dbad4332143ca6e570b8bd736a45e7" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390098 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390188 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390207 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q"] Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390273 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390308 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390336 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" event={"ID":"f47c9543-dee3-4e8e-8037-6dea7fba33f6","Type":"ContainerStarted","Data":"93e078c2e51d89d51d8f4af24ca80f2352de6e0e6bace8e92b72a50c53f63371"} Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390386 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390420 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390453 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390511 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390619 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390657 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:14:15.390845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.390759 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:14:15.393715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.393696 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:14:15.399538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.399516 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7"] Oct 31 23:14:15.487741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.487716 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af858a4d-b7eb-4d98-bba4-8e29f15b6791-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:15.560349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.560320 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:15.899481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.899385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:15.938424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:15.938394 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5"] Oct 31 23:14:16.356803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.356757 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" event={"ID":"56a5b524-ad27-4467-8e8d-4f57fdc73db2","Type":"ContainerStarted","Data":"a30e9caedd5203b4da6f045d5210b4703c474c3402d5b172da35a256992e887a"} Oct 31 23:14:16.358750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.358721 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" event={"ID":"dd1ae77f-51b8-4d9b-875b-daede18c8ab8","Type":"ContainerStarted","Data":"230c54538a0e43eb31c7e3298d818d1625fed8ed4315d01d9f7b297581a5922a"} Oct 31 23:14:16.358892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.358764 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" event={"ID":"dd1ae77f-51b8-4d9b-875b-daede18c8ab8","Type":"ContainerStarted","Data":"c681912c8bc40880e9330184c26363b0a6019fec0de6369056f568893b0ff11b"} Oct 31 23:14:16.360633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.360607 5043 generic.go:356] "Generic (PLEG): container finished" podID="f47c9543-dee3-4e8e-8037-6dea7fba33f6" containerID="591c4d54328c1ef5555703858cdf9b54e0cfd167c74b6b1e4f723fa72529442a" exitCode=0 Oct 31 23:14:16.360728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.360705 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" event={"ID":"f47c9543-dee3-4e8e-8037-6dea7fba33f6","Type":"ContainerDied","Data":"591c4d54328c1ef5555703858cdf9b54e0cfd167c74b6b1e4f723fa72529442a"} Oct 31 23:14:16.412588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.412521 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podStartSLOduration=7.412507075 podStartE2EDuration="7.412507075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:16.411468725 +0000 UTC m=+1482.641023137" watchObservedRunningTime="2026-10-31 23:14:16.412507075 +0000 UTC m=+1482.642061556" Oct 31 23:14:16.832887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:16.832831 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:17.232580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.231890 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:17.241150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.237707 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 23:14:17.241150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.237767 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 23:14:17.376245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.376156 5043 generic.go:356] "Generic (PLEG): container finished" podID="56a5b524-ad27-4467-8e8d-4f57fdc73db2" containerID="7720ba9aad150783629fecdc488a015f7396ceb840a97b36fdc8237c5f9f247f" exitCode=0 Oct 31 23:14:17.376400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.376369 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" event={"ID":"56a5b524-ad27-4467-8e8d-4f57fdc73db2","Type":"ContainerDied","Data":"7720ba9aad150783629fecdc488a015f7396ceb840a97b36fdc8237c5f9f247f"} Oct 31 23:14:17.595927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.595893 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:17.753570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.749753 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:17.836584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.813784 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca\") pod \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " Oct 31 23:14:17.837024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.836993 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca\") pod \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " Oct 31 23:14:17.837239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.837222 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7\" (UniqueName: \"kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7\") pod \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\" (UID: \"f47c9543-dee3-4e8e-8037-6dea7fba33f6\") " Oct 31 23:14:17.846047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.846012 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca" pod "f47c9543-dee3-4e8e-8037-6dea7fba33f6" (UID: "f47c9543-dee3-4e8e-8037-6dea7fba33f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:17.852992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.851472 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca" pod "f47c9543-dee3-4e8e-8037-6dea7fba33f6" (UID: "f47c9543-dee3-4e8e-8037-6dea7fba33f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:17.852992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.851505 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7" pod "f47c9543-dee3-4e8e-8037-6dea7fba33f6" (UID: "f47c9543-dee3-4e8e-8037-6dea7fba33f6"). InnerVolumeSpecName "kube-api-access-cdgs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:17.947281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.947193 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:17.947281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.947236 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9543-dee3-4e8e-8037-6dea7fba33f6-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:17.947281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:17.947252 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdgs7\" (UniqueName: \"kubernetes.io/projected/f47c9543-dee3-4e8e-8037-6dea7fba33f6-kube-api-access-cdgs7\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:18.240641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.233072 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:18.240641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.234205 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 23:14:18.240641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.234257 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 23:14:18.387826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.387792 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" Oct 31 23:14:18.387980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.387828 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q" event={"ID":"f47c9543-dee3-4e8e-8037-6dea7fba33f6","Type":"ContainerDied","Data":"93e078c2e51d89d51d8f4af24ca80f2352de6e0e6bace8e92b72a50c53f63371"} Oct 31 23:14:18.387980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.387883 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e078c2e51d89d51d8f4af24ca80f2352de6e0e6bace8e92b72a50c53f63371" Oct 31 23:14:18.656364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.656337 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:18.666265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.666237 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8\" (UniqueName: \"kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8\") pod \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " Oct 31 23:14:18.666392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.666333 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca\") pod \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " Oct 31 23:14:18.666605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.666588 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca\") pod \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\" (UID: \"56a5b524-ad27-4467-8e8d-4f57fdc73db2\") " Oct 31 23:14:18.666983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.666960 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca" pod "56a5b524-ad27-4467-8e8d-4f57fdc73db2" (UID: "56a5b524-ad27-4467-8e8d-4f57fdc73db2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:18.667353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.667334 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:18.668488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.668469 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8" pod "56a5b524-ad27-4467-8e8d-4f57fdc73db2" (UID: "56a5b524-ad27-4467-8e8d-4f57fdc73db2"). InnerVolumeSpecName "kube-api-access-76mr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:18.668598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.668491 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca" pod "56a5b524-ad27-4467-8e8d-4f57fdc73db2" (UID: "56a5b524-ad27-4467-8e8d-4f57fdc73db2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:18.768869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.768829 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76mr8\" (UniqueName: \"kubernetes.io/projected/56a5b524-ad27-4467-8e8d-4f57fdc73db2-kube-api-access-76mr8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:18.768869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.768870 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a5b524-ad27-4467-8e8d-4f57fdc73db2-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:18.900107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:18.899998 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:19.233316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:19.232922 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 23:14:19.233316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:19.232985 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 23:14:19.408316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:19.408270 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" event={"ID":"56a5b524-ad27-4467-8e8d-4f57fdc73db2","Type":"ContainerDied","Data":"a30e9caedd5203b4da6f045d5210b4703c474c3402d5b172da35a256992e887a"} Oct 31 23:14:19.408822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:19.408374 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a30e9caedd5203b4da6f045d5210b4703c474c3402d5b172da35a256992e887a" Oct 31 23:14:19.408822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:19.408490 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5" Oct 31 23:14:19.640583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:19.639763 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:20.241619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:20.238995 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 23:14:20.241619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:20.239075 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 23:14:20.376651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:20.376600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:20.636629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:20.628100 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:21.093832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.093798 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:21.233929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.233891 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 23:14:21.234749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.234713 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 23:14:21.387658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.387538 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:21.821045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.821005 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v"] Oct 31 23:14:21.822342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822305 5043 cpu_manager.go:414] "containerMap: removing container" podUID="af858a4d-b7eb-4d98-bba4-8e29f15b6791" containerName="revoke-keys" Oct 31 23:14:21.822342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822332 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af858a4d-b7eb-4d98-bba4-8e29f15b6791" containerName="revoke-keys" Oct 31 23:14:21.822566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822372 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f47c9543-dee3-4e8e-8037-6dea7fba33f6" containerName="revoke-keys" Oct 31 23:14:21.822566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822381 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47c9543-dee3-4e8e-8037-6dea7fba33f6" containerName="revoke-keys" Oct 31 23:14:21.822566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822396 5043 cpu_manager.go:414] "containerMap: removing container" podUID="56a5b524-ad27-4467-8e8d-4f57fdc73db2" containerName="revoke-keys" Oct 31 23:14:21.822566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822404 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a5b524-ad27-4467-8e8d-4f57fdc73db2" containerName="revoke-keys" Oct 31 23:14:21.822800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822616 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47c9543-dee3-4e8e-8037-6dea7fba33f6" containerName="revoke-keys" Oct 31 23:14:21.822800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822631 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a5b524-ad27-4467-8e8d-4f57fdc73db2" containerName="revoke-keys" Oct 31 23:14:21.822800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.822657 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="af858a4d-b7eb-4d98-bba4-8e29f15b6791" containerName="revoke-keys" Oct 31 23:14:21.827643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.827612 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.843467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.841540 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:14:21.843467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.841680 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:14:21.843467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.841732 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:14:21.843467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.842235 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:14:21.882618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.869656 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v"] Oct 31 23:14:21.895139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.895018 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvft\" (UniqueName: \"kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.895139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.895089 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.895339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.895145 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.996831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.996793 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvft\" (UniqueName: \"kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.997007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.996867 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.997007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.996928 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.997674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.997646 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:21.999377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:21.999352 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:22.009665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.009633 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvft\" (UniqueName: \"kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft\") pod \"maas-api-revoke-sub-66cc15d5-87j8v\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:22.184350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.184266 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:22.254584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.240034 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 23:14:22.254584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.240096 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 23:14:22.254584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.242520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:22.561761 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:22.551451 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b9fba4_ce9b_4012_a45d_ce2bb428079c.slice/crio-cce544de9c071996ce27c50e28e1ffd026d2c544a7fb4014c5e46e6c4674ba87 WatchSource:0}: Error finding container cce544de9c071996ce27c50e28e1ffd026d2c544a7fb4014c5e46e6c4674ba87: Status 404 returned error can't find the container with id cce544de9c071996ce27c50e28e1ffd026d2c544a7fb4014c5e46e6c4674ba87 Oct 31 23:14:22.561761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.552431 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v"] Oct 31 23:14:22.572826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:22.572788 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" event={"ID":"a0b9fba4-ce9b-4012-a45d-ce2bb428079c","Type":"ContainerStarted","Data":"cce544de9c071996ce27c50e28e1ffd026d2c544a7fb4014c5e46e6c4674ba87"} Oct 31 23:14:23.022015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.021983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:23.326991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.325430 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:23.326991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.326824 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:23.327824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.327802 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:14:23.459380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.459333 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl"] Oct 31 23:14:23.471459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.471388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:23.473071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.472382 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl"] Oct 31 23:14:23.473071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.472496 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.560463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.558878 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.560463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.558988 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.560463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.559060 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fhv\" (UniqueName: \"kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.604235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.603653 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" event={"ID":"a0b9fba4-ce9b-4012-a45d-ce2bb428079c","Type":"ContainerStarted","Data":"97a6008ba5f7ac6dce263300b8d35dfaf62abe78dc52aeeb4e82f00b3f351c06"} Oct 31 23:14:23.631146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.631105 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:23.634769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.634722 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" podStartSLOduration=2.634707562 podStartE2EDuration="2.634707562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:23.625814772 +0000 UTC m=+1489.855369185" watchObservedRunningTime="2026-10-31 23:14:23.634707562 +0000 UTC m=+1489.864261973" Oct 31 23:14:23.661190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.661162 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.661350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.661247 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.661350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.661295 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fhv\" (UniqueName: \"kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.662052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.662022 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.663720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.663687 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.673408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.673387 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fhv\" (UniqueName: \"kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv\") pod \"maas-api-revoke-sub-473e18ab-bhjbl\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.776383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.775441 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:23.792735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.792708 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:23.840782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.840749 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b"] Oct 31 23:14:23.847903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.847862 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:23.857034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.856949 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b"] Oct 31 23:14:23.996526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:23.996488 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:24.018858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.018809 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.022448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.022414 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vspw\" (UniqueName: \"kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.022821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.022500 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.137276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.137162 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vspw\" (UniqueName: \"kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.137930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.137902 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.138231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.138209 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.139289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.139250 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.143435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.142377 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.166808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.166761 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vspw\" (UniqueName: \"kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw\") pod \"maas-api-revoke-sub-a0083906-znt5b\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.175203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.175178 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:24.198491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.197295 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl"] Oct 31 23:14:24.240682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.240655 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:24.328650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.328411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:24.337979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.334054 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv"] Oct 31 23:14:24.374349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.374317 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv"] Oct 31 23:14:24.374711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.374636 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.407072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.407009 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b2c6553b-openshift-default-dockercfg-9pd6s\"" Oct 31 23:14:24.704472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.596353 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:24.705077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.665734 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.705290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.705270 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdlxf\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.705484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.705459 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.705710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.705680 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.711866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.705944 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.722782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.722747 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.739149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.739116 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.753303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.751100 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.763741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.763701 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.800019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.799983 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b"] Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:24.869168 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ded6680_9832_4e62_b762_f0665c04996a.slice/crio-d6361d5fad0662e8950ea8c88bcfff95197e8aaed34caee374ac0c8bcdc533a6 WatchSource:0}: Error finding container d6361d5fad0662e8950ea8c88bcfff95197e8aaed34caee374ac0c8bcdc533a6: Status 404 returned error can't find the container with id d6361d5fad0662e8950ea8c88bcfff95197e8aaed34caee374ac0c8bcdc533a6 Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.869581 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" event={"ID":"8bf025a7-87b6-4a6c-9a76-44e61035e408","Type":"ContainerStarted","Data":"c9f46011fd784d0904cb927e8831892efc8a61cb414fec2c60637a7d43a89f71"} Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.870018 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.880463 5043 generic.go:356] "Generic (PLEG): container finished" podID="a0b9fba4-ce9b-4012-a45d-ce2bb428079c" containerID="97a6008ba5f7ac6dce263300b8d35dfaf62abe78dc52aeeb4e82f00b3f351c06" exitCode=0 Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.882273 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" event={"ID":"a0b9fba4-ce9b-4012-a45d-ce2bb428079c","Type":"ContainerDied","Data":"97a6008ba5f7ac6dce263300b8d35dfaf62abe78dc52aeeb4e82f00b3f351c06"} Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884539 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884596 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884622 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884682 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884794 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884846 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdlxf\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884907 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.884945 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.888648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.885040 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.969814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.969005 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.976243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.973010 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdlxf\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.976243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.973692 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.976243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.975052 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.976243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.975242 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.976243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.975500 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.976243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.975667 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.978335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.978288 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.978677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.978472 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token\") pod \"e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:24.996802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:24.996778 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:25.153572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.142346 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:25.478700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.478664 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv"] Oct 31 23:14:25.481086 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:25.481063 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9209e0_2e53_48cb_9855_d711a23ad9e5.slice/crio-e283a2cd870a31cb860814e17a5bf197d13be1af46799e2fc0c45b99db7f60df WatchSource:0}: Error finding container e283a2cd870a31cb860814e17a5bf197d13be1af46799e2fc0c45b99db7f60df: Status 404 returned error can't find the container with id e283a2cd870a31cb860814e17a5bf197d13be1af46799e2fc0c45b99db7f60df Oct 31 23:14:25.488343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.484584 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:25.488343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.484720 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:25.488343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.484798 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:25.898794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.898705 5043 generic.go:356] "Generic (PLEG): container finished" podID="8bf025a7-87b6-4a6c-9a76-44e61035e408" containerID="d54ffc6e9ecda7b1d91616d4312030722ab605b99e491cb4995a096e9ffd1ea6" exitCode=0 Oct 31 23:14:25.898962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.898860 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" event={"ID":"8bf025a7-87b6-4a6c-9a76-44e61035e408","Type":"ContainerDied","Data":"d54ffc6e9ecda7b1d91616d4312030722ab605b99e491cb4995a096e9ffd1ea6"} Oct 31 23:14:25.905558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.905453 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" event={"ID":"3d9209e0-2e53-48cb-9855-d711a23ad9e5","Type":"ContainerStarted","Data":"d5b88374f2d00a8bdda3dc21ec7d90765349a8c790b50f263735665dd191ecd7"} Oct 31 23:14:25.905558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.905522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" event={"ID":"3d9209e0-2e53-48cb-9855-d711a23ad9e5","Type":"ContainerStarted","Data":"e283a2cd870a31cb860814e17a5bf197d13be1af46799e2fc0c45b99db7f60df"} Oct 31 23:14:25.907309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.907274 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" event={"ID":"1ded6680-9832-4e62-b762-f0665c04996a","Type":"ContainerStarted","Data":"d170adf03be7977f19cbef8478742288e051fbd4379b2298094b536855160750"} Oct 31 23:14:25.907463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.907336 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" event={"ID":"1ded6680-9832-4e62-b762-f0665c04996a","Type":"ContainerStarted","Data":"d6361d5fad0662e8950ea8c88bcfff95197e8aaed34caee374ac0c8bcdc533a6"} Oct 31 23:14:25.955863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.955591 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" podStartSLOduration=2.955571635 podStartE2EDuration="2.955571635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:25.947062662 +0000 UTC m=+1492.176617073" watchObservedRunningTime="2026-10-31 23:14:25.955571635 +0000 UTC m=+1492.185126048" Oct 31 23:14:25.974913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:25.974861 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" podStartSLOduration=1.974848218 podStartE2EDuration="1.974848218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:25.97032794 +0000 UTC m=+1492.199882385" watchObservedRunningTime="2026-10-31 23:14:25.974848218 +0000 UTC m=+1492.204402629" Oct 31 23:14:26.003351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.001645 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:26.005047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.005015 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:26.008365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.008344 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:26.193539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.193463 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:26.377881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.377845 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca\") pod \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " Oct 31 23:14:26.378118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.378104 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca\") pod \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " Oct 31 23:14:26.378207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.378167 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft\" (UniqueName: \"kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft\") pod \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\" (UID: \"a0b9fba4-ce9b-4012-a45d-ce2bb428079c\") " Oct 31 23:14:26.378474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.378451 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca" pod "a0b9fba4-ce9b-4012-a45d-ce2bb428079c" (UID: "a0b9fba4-ce9b-4012-a45d-ce2bb428079c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:26.378783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.378763 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:26.380062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.380028 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca" pod "a0b9fba4-ce9b-4012-a45d-ce2bb428079c" (UID: "a0b9fba4-ce9b-4012-a45d-ce2bb428079c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:26.380297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.380268 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft" pod "a0b9fba4-ce9b-4012-a45d-ce2bb428079c" (UID: "a0b9fba4-ce9b-4012-a45d-ce2bb428079c"). InnerVolumeSpecName "kube-api-access-6wvft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:26.480605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.480483 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:26.480605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.480520 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvft\" (UniqueName: \"kubernetes.io/projected/a0b9fba4-ce9b-4012-a45d-ce2bb428079c-kube-api-access-6wvft\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:26.919992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.919943 5043 generic.go:356] "Generic (PLEG): container finished" podID="1ded6680-9832-4e62-b762-f0665c04996a" containerID="d170adf03be7977f19cbef8478742288e051fbd4379b2298094b536855160750" exitCode=0 Oct 31 23:14:26.920182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.920024 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" event={"ID":"1ded6680-9832-4e62-b762-f0665c04996a","Type":"ContainerDied","Data":"d170adf03be7977f19cbef8478742288e051fbd4379b2298094b536855160750"} Oct 31 23:14:26.921790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.921768 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" Oct 31 23:14:26.921930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.921793 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v" event={"ID":"a0b9fba4-ce9b-4012-a45d-ce2bb428079c","Type":"ContainerDied","Data":"cce544de9c071996ce27c50e28e1ffd026d2c544a7fb4014c5e46e6c4674ba87"} Oct 31 23:14:26.921930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.921841 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce544de9c071996ce27c50e28e1ffd026d2c544a7fb4014c5e46e6c4674ba87" Oct 31 23:14:26.922322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.922298 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:26.923867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:26.923790 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:14:27.205277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.205249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:27.207571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.207534 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:27.300444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.300412 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca\") pod \"8bf025a7-87b6-4a6c-9a76-44e61035e408\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " Oct 31 23:14:27.300741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.300725 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca\") pod \"8bf025a7-87b6-4a6c-9a76-44e61035e408\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " Oct 31 23:14:27.300803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.300755 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca" pod "8bf025a7-87b6-4a6c-9a76-44e61035e408" (UID: "8bf025a7-87b6-4a6c-9a76-44e61035e408"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:27.300803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.300767 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv\" (UniqueName: \"kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv\") pod \"8bf025a7-87b6-4a6c-9a76-44e61035e408\" (UID: \"8bf025a7-87b6-4a6c-9a76-44e61035e408\") " Oct 31 23:14:27.301581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.301348 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:27.302879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.302854 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca" pod "8bf025a7-87b6-4a6c-9a76-44e61035e408" (UID: "8bf025a7-87b6-4a6c-9a76-44e61035e408"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:27.302971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.302888 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv" pod "8bf025a7-87b6-4a6c-9a76-44e61035e408" (UID: "8bf025a7-87b6-4a6c-9a76-44e61035e408"). InnerVolumeSpecName "kube-api-access-w2fhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:27.403310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.403283 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf025a7-87b6-4a6c-9a76-44e61035e408-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:27.403310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.403308 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fhv\" (UniqueName: \"kubernetes.io/projected/8bf025a7-87b6-4a6c-9a76-44e61035e408-kube-api-access-w2fhv\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:27.520973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.520928 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:27.672285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.672254 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:27.817653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.817563 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:27.934563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.934524 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" Oct 31 23:14:27.939506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.939463 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl" event={"ID":"8bf025a7-87b6-4a6c-9a76-44e61035e408","Type":"ContainerDied","Data":"c9f46011fd784d0904cb927e8831892efc8a61cb414fec2c60637a7d43a89f71"} Oct 31 23:14:27.939629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:27.939528 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9f46011fd784d0904cb927e8831892efc8a61cb414fec2c60637a7d43a89f71" Oct 31 23:14:28.119005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.112849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:28.185788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.181809 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.268523 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4"] Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269645 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1ded6680-9832-4e62-b762-f0665c04996a" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269665 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ded6680-9832-4e62-b762-f0665c04996a" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269682 5043 cpu_manager.go:414] "containerMap: removing container" podUID="8bf025a7-87b6-4a6c-9a76-44e61035e408" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269689 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf025a7-87b6-4a6c-9a76-44e61035e408" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269735 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a0b9fba4-ce9b-4012-a45d-ce2bb428079c" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269743 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b9fba4-ce9b-4012-a45d-ce2bb428079c" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269947 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ded6680-9832-4e62-b762-f0665c04996a" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269960 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf025a7-87b6-4a6c-9a76-44e61035e408" containerName="revoke-keys" Oct 31 23:14:28.273609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.269974 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b9fba4-ce9b-4012-a45d-ce2bb428079c" containerName="revoke-keys" Oct 31 23:14:28.276565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.274899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:28.276565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.275408 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.281828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.281665 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ed6117ae-dockercfg-mfksb\"" Oct 31 23:14:28.281828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.281766 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ed6117ae\"" Oct 31 23:14:28.294311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.294276 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4"] Oct 31 23:14:28.324566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.322347 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca\") pod \"1ded6680-9832-4e62-b762-f0665c04996a\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " Oct 31 23:14:28.324566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.322465 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca\") pod \"1ded6680-9832-4e62-b762-f0665c04996a\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " Oct 31 23:14:28.324566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.322877 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw\" (UniqueName: \"kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw\") pod \"1ded6680-9832-4e62-b762-f0665c04996a\" (UID: \"1ded6680-9832-4e62-b762-f0665c04996a\") " Oct 31 23:14:28.324566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.324319 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca" pod "1ded6680-9832-4e62-b762-f0665c04996a" (UID: "1ded6680-9832-4e62-b762-f0665c04996a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:28.335088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.335047 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw" pod "1ded6680-9832-4e62-b762-f0665c04996a" (UID: "1ded6680-9832-4e62-b762-f0665c04996a"). InnerVolumeSpecName "kube-api-access-8vspw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:28.336954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.336921 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca" pod "1ded6680-9832-4e62-b762-f0665c04996a" (UID: "1ded6680-9832-4e62-b762-f0665c04996a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:28.435589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.433624 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pslwg\" (UniqueName: \"kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg\") pod \"payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.435589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.433760 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.435589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.433897 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vspw\" (UniqueName: \"kubernetes.io/projected/1ded6680-9832-4e62-b762-f0665c04996a-kube-api-access-8vspw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:28.435589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.433914 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:28.481265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.481168 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ded6680-9832-4e62-b762-f0665c04996a-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:28.586953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.584428 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pslwg\" (UniqueName: \"kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg\") pod \"payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.586953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.584616 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.586953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.585963 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.606526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.606356 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pslwg\" (UniqueName: \"kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg\") pod \"payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.627015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.626930 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:28.777831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.777793 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:28.895770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.895685 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:28.942309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.942275 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:28.953491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.953462 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" Oct 31 23:14:28.953662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.953487 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b" event={"ID":"1ded6680-9832-4e62-b762-f0665c04996a","Type":"ContainerDied","Data":"d6361d5fad0662e8950ea8c88bcfff95197e8aaed34caee374ac0c8bcdc533a6"} Oct 31 23:14:28.953662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.953598 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6361d5fad0662e8950ea8c88bcfff95197e8aaed34caee374ac0c8bcdc533a6" Oct 31 23:14:28.966793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.966761 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7"] Oct 31 23:14:28.974971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.974941 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:28.979331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.979281 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b2c6553b\"" Oct 31 23:14:28.986380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.985274 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7"] Oct 31 23:14:28.998168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:28.998146 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd"] Oct 31 23:14:29.003783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.003761 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.021686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.012648 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b2c6553b-dockercfg-k6w6b\"" Oct 31 23:14:29.024080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.024052 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd"] Oct 31 23:14:29.091485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.091447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:29.095706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.094215 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.095706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.094321 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr\") pod \"payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.241699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.241135 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr\") pod \"payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.287271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.287228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:29.301634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.292705 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.301634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.292916 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkq9\" (UniqueName: \"kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9\") pod \"payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.301634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.293056 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.301634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.294101 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.301634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.297339 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr\") pod \"payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.418568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.395040 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkq9\" (UniqueName: \"kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9\") pod \"payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.435567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.434127 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.444011 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:29.443906 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5adf9c0_7d39_422e_8f9b_e64c4572477a.slice/crio-4955cd987fb043633520f011e6c07b4d700d2a779137219a3684df720c552dbd WatchSource:0}: Error finding container 4955cd987fb043633520f011e6c07b4d700d2a779137219a3684df720c552dbd: Status 404 returned error can't find the container with id 4955cd987fb043633520f011e6c07b4d700d2a779137219a3684df720c552dbd Oct 31 23:14:29.457087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.452364 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4"] Oct 31 23:14:29.478574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.468502 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.478574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.474301 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkq9\" (UniqueName: \"kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9\") pod \"payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.595311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.595283 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:29.656389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.656139 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:29.696595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.693419 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc"] Oct 31 23:14:29.754623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.754582 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc"] Oct 31 23:14:29.754981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.754950 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.773664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.772689 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ac5ff20f-openshift-default-dockercfg-zz9vc\"" Oct 31 23:14:29.842626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.826125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:29.898582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.898526 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbn6\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.899006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.898977 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.899478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.899451 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.900300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.900276 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.906484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.906463 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.906754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.906704 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.906948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.906930 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.907149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.907115 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:29.907321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:29.907286 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.014008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.013958 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.014362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.014314 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.019530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.019495 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.019733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.019712 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.019940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.019924 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbn6\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.020079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.020061 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.020418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.020375 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.024359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.020590 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.024562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.024527 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.057240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.056385 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.057240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.056865 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.066566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.061380 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.066566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.062118 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.098219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.098164 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.100908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.100220 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.102292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.102046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:30.135459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.135368 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.147279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.147218 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.150402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.149990 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbn6\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6\") pod \"e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.171601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.171559 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" event={"ID":"c5adf9c0-7d39-422e-8f9b-e64c4572477a","Type":"ContainerStarted","Data":"df0c471c5ce9e9da0e829ab95bf304e1556f8e13f5011d01bc8a85879b1ef832"} Oct 31 23:14:30.171776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.171630 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" event={"ID":"c5adf9c0-7d39-422e-8f9b-e64c4572477a","Type":"ContainerStarted","Data":"4955cd987fb043633520f011e6c07b4d700d2a779137219a3684df720c552dbd"} Oct 31 23:14:30.172891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.172863 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:30.215614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.215491 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" podStartSLOduration=2.215475394 podStartE2EDuration="2.215475394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:30.19972008 +0000 UTC m=+1496.429274491" watchObservedRunningTime="2026-10-31 23:14:30.215475394 +0000 UTC m=+1496.445029819" Oct 31 23:14:30.216684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.216655 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:30.266237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.266202 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd"] Oct 31 23:14:30.270818 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:30.270790 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66aa7ac3_0ebb_4b0d_ab84_33d8b51b3036.slice/crio-37dbe9b7b186edc30a1fb4405eefae88612bba64c4139400946d7aaecc4f0b6b WatchSource:0}: Error finding container 37dbe9b7b186edc30a1fb4405eefae88612bba64c4139400946d7aaecc4f0b6b: Status 404 returned error can't find the container with id 37dbe9b7b186edc30a1fb4405eefae88612bba64c4139400946d7aaecc4f0b6b Oct 31 23:14:30.272116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.271980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:30.321600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.321523 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7"] Oct 31 23:14:30.348376 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:30.328485 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97551e11_91d7_421c_aad0_aee5c3c8b8b2.slice/crio-1d559f1523a92b4815b3675f4663e95fcf60ff1d0eceeca72bbfb74838a91333 WatchSource:0}: Error finding container 1d559f1523a92b4815b3675f4663e95fcf60ff1d0eceeca72bbfb74838a91333: Status 404 returned error can't find the container with id 1d559f1523a92b4815b3675f4663e95fcf60ff1d0eceeca72bbfb74838a91333 Oct 31 23:14:30.754910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.754871 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc"] Oct 31 23:14:30.822675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.820745 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:30.822675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.820833 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:30.822675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.820897 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:14:30.905689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:30.873861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:31.259291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.259250 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" event={"ID":"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036","Type":"ContainerStarted","Data":"44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50"} Oct 31 23:14:31.266973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.266927 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" event={"ID":"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036","Type":"ContainerStarted","Data":"37dbe9b7b186edc30a1fb4405eefae88612bba64c4139400946d7aaecc4f0b6b"} Oct 31 23:14:31.269641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.269612 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:31.306098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.306053 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" event={"ID":"ff91cb2a-b941-422f-b17b-549ec736d03f","Type":"ContainerStarted","Data":"4c9c5441394beb6e9050e7dd89f746fb4674569441d53d2a734148cbacba3c81"} Oct 31 23:14:31.311503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.311473 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" event={"ID":"97551e11-91d7-421c-aad0-aee5c3c8b8b2","Type":"ContainerStarted","Data":"407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9"} Oct 31 23:14:31.311737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.311721 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:31.311908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.311887 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" event={"ID":"97551e11-91d7-421c-aad0-aee5c3c8b8b2","Type":"ContainerStarted","Data":"1d559f1523a92b4815b3675f4663e95fcf60ff1d0eceeca72bbfb74838a91333"} Oct 31 23:14:31.338951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.337576 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" podStartSLOduration=3.315436538 podStartE2EDuration="3.315436538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:31.294952993 +0000 UTC m=+1497.524507404" watchObservedRunningTime="2026-10-31 23:14:31.315436538 +0000 UTC m=+1497.544990949" Oct 31 23:14:31.400380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:31.395894 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" podStartSLOduration=3.395874269 podStartE2EDuration="3.395874269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:31.348603743 +0000 UTC m=+1497.578158149" watchObservedRunningTime="2026-10-31 23:14:31.395874269 +0000 UTC m=+1497.625428679" Oct 31 23:14:32.476315 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.476280 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" event={"ID":"ff91cb2a-b941-422f-b17b-549ec736d03f","Type":"ContainerStarted","Data":"dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244"} Oct 31 23:14:32.578588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.578498 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" podStartSLOduration=3.578474619 podStartE2EDuration="3.578474619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:32.512013573 +0000 UTC m=+1498.741567983" watchObservedRunningTime="2026-10-31 23:14:32.578474619 +0000 UTC m=+1498.808029030" Oct 31 23:14:32.688710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.688664 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:32.750510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.750465 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:32.787693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.787642 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:33.018628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.933987 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.029364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.949776 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:14:33.039583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.949871 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.059718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.979838 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.060305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.979895 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:33.071590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:32.979916 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:33.122539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.122430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:33.271673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.154068 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.307587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.298837 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:33.412940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.412898 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:33.492702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.492657 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:33.875686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.709933 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:33.889012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.772344 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:33.902865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.772376 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:14:33.903595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.877313 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:33.903822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.903802 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:14:33.909370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.909348 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:14:33.924642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:33.924596 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.006137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.006098 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.024572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.011430 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:34.024856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.024820 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.508324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.508281 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded" Oct 31 23:14:34.557682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.557631 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:34.682571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.655591 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:14:34.733533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.712044 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:14:34.735693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.734510 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:34.848113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.848020 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:34.848367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.848336 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:34.875572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.850497 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:34.878118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.875973 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:35.225153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.899337 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:35.646370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.929719 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.647144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:34.969160 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 31 23:14:35.647342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.031708 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:35.647508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.405445 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.647726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.405516 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:35.684570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.405397 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:35.684570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.670519 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:14:35.684570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.670949 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded" Oct 31 23:14:35.713777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.713734 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.769776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.769725 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:14:35.904078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.903682 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.904464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.903766 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:35.904694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.903794 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 23:14:35.930630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:35.930579 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.056756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.056705 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:36.057157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.057021 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.057408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.057380 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.098570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.090944 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:36.098570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.091016 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.177623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.140581 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.320989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.241330 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:36.426570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.419157 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.703792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.703757 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:36.712287 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.712236 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.756990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.756942 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:36.805662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.757247 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:14:36.805978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.805943 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:36.806149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.806129 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:36.806282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.806264 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:14:36.806427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.806407 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:37.247984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.958824 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:37.255571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:36.960234 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:37.277860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.277815 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:37.302573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.278138 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:37.401573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.326819 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:37.401573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.008321 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:14:37.401573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.326094 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 23:14:37.401573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.280117 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 31 23:14:37.401573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.325050 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:14:37.401573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.342622 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:37.426594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.423775 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:37.571909 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:14:37.571822 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.618s" Oct 31 23:14:37.806072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:37.806011 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:38.118568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.080608 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:38.543861 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.543811 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:38.571578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.569193 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:38.571578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.570487 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:38.571578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.570669 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:14:38.571578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.570931 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:14:38.603942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.590276 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:14:38.664792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.643680 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:38.664792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.644495 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:14:38.664792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.645279 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:14:38.779564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.769471 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:14:38.805001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.804906 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:14:38.805477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.805450 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:14:38.823561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:38.806374 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded" Oct 31 23:14:39.014307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.003977 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:39.015436 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.015406 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:14:39.015697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.015682 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:14:39.015855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.015843 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:14:39.015964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.015955 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:14:39.016084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.016074 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:14:39.016191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.016180 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:14:39.016290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.016279 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw"] Oct 31 23:14:39.055604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.052455 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:14:39.055604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.052572 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.055604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.054231 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:14:39.060562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.057311 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl"] Oct 31 23:14:39.060736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.060581 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:14:39.060736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.060643 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:14:39.060736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.060730 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:14:39.060926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.060803 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:14:39.102606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102564 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw"] Oct 31 23:14:39.102751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102672 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:14:39.102751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102720 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl"] Oct 31 23:14:39.102869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102785 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:14:39.102869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102843 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:14:39.102966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102884 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:14:39.102966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.102946 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:14:39.103057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103003 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:14:39.103057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103043 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:14:39.103154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103047 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvr6\" (UniqueName: \"kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.103154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103092 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:14:39.103154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103122 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:14:39.103154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103137 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103167 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103184 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103200 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103244 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103271 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103312 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103324 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.103354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103342 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:14:39.103741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.103442 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.104717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.104674 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.114631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.114610 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:14:39.132711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.132673 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:14:39.211869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.211147 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvr6\" (UniqueName: \"kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.211869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.211286 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.211869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.211439 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.211869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.211476 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.211869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.211598 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.211869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.211770 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s87nx\" (UniqueName: \"kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.222478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.218283 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.250600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.237378 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.288193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.288148 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:14:39.315998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.313680 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:14:39.327575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.327189 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.327575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.327413 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.327846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.327639 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s87nx\" (UniqueName: \"kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.329264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.329135 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.359019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.358965 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:39.443625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.430627 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.443625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.431662 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvr6\" (UniqueName: \"kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6\") pod \"maas-api-revoke-sub-6f08a863-5g4jw\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.443625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.434735 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s87nx\" (UniqueName: \"kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx\") pod \"maas-api-revoke-sub-31fd3132-v8mbl\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.477381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.474796 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:39.477381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.475047 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:39.524314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:39.522194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:40.169584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:40.160623 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:40.357902 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:40.356913 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c58484_9578_4663_813f_ae4b5c8490f7.slice/crio-18eb13ddb5fd75f74a3ecf0ea80180fac1896227ee54387c992eaf6a88ea5239 WatchSource:0}: Error finding container 18eb13ddb5fd75f74a3ecf0ea80180fac1896227ee54387c992eaf6a88ea5239: Status 404 returned error can't find the container with id 18eb13ddb5fd75f74a3ecf0ea80180fac1896227ee54387c992eaf6a88ea5239 Oct 31 23:14:40.374326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:40.373624 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw"] Oct 31 23:14:40.383635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:40.383525 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl"] Oct 31 23:14:40.603615 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:14:40.600239 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:14:40.689503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:40.689455 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:40.864578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:40.862889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:41.046360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.046328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:41.373400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.373351 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" event={"ID":"28d68379-d5b2-4caf-af89-9a293fea71b7","Type":"ContainerStarted","Data":"2e8a1d64c32b3211512abbc141fd85532390fb08da123af970a805af5a6f0692"} Oct 31 23:14:41.374428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.374404 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" event={"ID":"28d68379-d5b2-4caf-af89-9a293fea71b7","Type":"ContainerStarted","Data":"6267976f051eb6045ec6b80a00d78947d38d000021e798c1df453f426ce834e3"} Oct 31 23:14:41.375363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.375340 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" event={"ID":"52c58484-9578-4663-813f-ae4b5c8490f7","Type":"ContainerStarted","Data":"ea5e2074eae85308b0fb671db43f7d3af0f142f1cd30c751b8dcb25c66300f6a"} Oct 31 23:14:41.375504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.375492 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" event={"ID":"52c58484-9578-4663-813f-ae4b5c8490f7","Type":"ContainerStarted","Data":"18eb13ddb5fd75f74a3ecf0ea80180fac1896227ee54387c992eaf6a88ea5239"} Oct 31 23:14:41.395927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.395882 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" podStartSLOduration=6.395870792 podStartE2EDuration="6.395870792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:41.393759065 +0000 UTC m=+1507.623313477" watchObservedRunningTime="2026-10-31 23:14:41.395870792 +0000 UTC m=+1507.625425198" Oct 31 23:14:41.420408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.420356 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" podStartSLOduration=6.420340042 podStartE2EDuration="6.420340042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:41.412709486 +0000 UTC m=+1507.642263893" watchObservedRunningTime="2026-10-31 23:14:41.420340042 +0000 UTC m=+1507.649894454" Oct 31 23:14:41.432526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:41.432489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:42.178278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.178002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:42.386935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.386903 5043 generic.go:356] "Generic (PLEG): container finished" podID="28d68379-d5b2-4caf-af89-9a293fea71b7" containerID="2e8a1d64c32b3211512abbc141fd85532390fb08da123af970a805af5a6f0692" exitCode=0 Oct 31 23:14:42.387484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.386950 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" event={"ID":"28d68379-d5b2-4caf-af89-9a293fea71b7","Type":"ContainerDied","Data":"2e8a1d64c32b3211512abbc141fd85532390fb08da123af970a805af5a6f0692"} Oct 31 23:14:42.389367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.389343 5043 generic.go:356] "Generic (PLEG): container finished" podID="52c58484-9578-4663-813f-ae4b5c8490f7" containerID="ea5e2074eae85308b0fb671db43f7d3af0f142f1cd30c751b8dcb25c66300f6a" exitCode=0 Oct 31 23:14:42.389505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.389407 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" event={"ID":"52c58484-9578-4663-813f-ae4b5c8490f7","Type":"ContainerDied","Data":"ea5e2074eae85308b0fb671db43f7d3af0f142f1cd30c751b8dcb25c66300f6a"} Oct 31 23:14:42.480831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.480759 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:14:42.488866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.488845 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:14:42.923143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:42.923112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:43.588879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.588756 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:43.644596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.637870 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:43.731570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.728035 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx\" (UniqueName: \"kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx\") pod \"52c58484-9578-4663-813f-ae4b5c8490f7\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " Oct 31 23:14:43.731570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.729465 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca\") pod \"52c58484-9578-4663-813f-ae4b5c8490f7\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " Oct 31 23:14:43.731570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.731215 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca" pod "52c58484-9578-4663-813f-ae4b5c8490f7" (UID: "52c58484-9578-4663-813f-ae4b5c8490f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:43.731890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.731659 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca\") pod \"52c58484-9578-4663-813f-ae4b5c8490f7\" (UID: \"52c58484-9578-4663-813f-ae4b5c8490f7\") " Oct 31 23:14:43.733564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.732564 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:43.733938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.733903 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx" pod "52c58484-9578-4663-813f-ae4b5c8490f7" (UID: "52c58484-9578-4663-813f-ae4b5c8490f7"). InnerVolumeSpecName "kube-api-access-s87nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:43.735828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.735172 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca" pod "52c58484-9578-4663-813f-ae4b5c8490f7" (UID: "52c58484-9578-4663-813f-ae4b5c8490f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:43.770043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.770018 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:43.834098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.834059 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca\") pod \"28d68379-d5b2-4caf-af89-9a293fea71b7\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " Oct 31 23:14:43.834345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.834112 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca\") pod \"28d68379-d5b2-4caf-af89-9a293fea71b7\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " Oct 31 23:14:43.834345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.834192 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6\" (UniqueName: \"kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6\") pod \"28d68379-d5b2-4caf-af89-9a293fea71b7\" (UID: \"28d68379-d5b2-4caf-af89-9a293fea71b7\") " Oct 31 23:14:43.835106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.835066 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s87nx\" (UniqueName: \"kubernetes.io/projected/52c58484-9578-4663-813f-ae4b5c8490f7-kube-api-access-s87nx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:43.835106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.835099 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c58484-9578-4663-813f-ae4b5c8490f7-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:43.835623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.835598 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca" pod "28d68379-d5b2-4caf-af89-9a293fea71b7" (UID: "28d68379-d5b2-4caf-af89-9a293fea71b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:43.837796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.837774 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca" pod "28d68379-d5b2-4caf-af89-9a293fea71b7" (UID: "28d68379-d5b2-4caf-af89-9a293fea71b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:43.839326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.839305 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6" pod "28d68379-d5b2-4caf-af89-9a293fea71b7" (UID: "28d68379-d5b2-4caf-af89-9a293fea71b7"). InnerVolumeSpecName "kube-api-access-hqvr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:43.894926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.894897 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:14:43.955046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.954703 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:43.955046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.954743 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d68379-d5b2-4caf-af89-9a293fea71b7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:43.955046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:43.954762 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvr6\" (UniqueName: \"kubernetes.io/projected/28d68379-d5b2-4caf-af89-9a293fea71b7-kube-api-access-hqvr6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:44.249391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.249357 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:44.411833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.411807 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" Oct 31 23:14:44.411833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.411823 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl" event={"ID":"52c58484-9578-4663-813f-ae4b5c8490f7","Type":"ContainerDied","Data":"18eb13ddb5fd75f74a3ecf0ea80180fac1896227ee54387c992eaf6a88ea5239"} Oct 31 23:14:44.412045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.411870 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18eb13ddb5fd75f74a3ecf0ea80180fac1896227ee54387c992eaf6a88ea5239" Oct 31 23:14:44.414086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.414056 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" event={"ID":"28d68379-d5b2-4caf-af89-9a293fea71b7","Type":"ContainerDied","Data":"6267976f051eb6045ec6b80a00d78947d38d000021e798c1df453f426ce834e3"} Oct 31 23:14:44.414219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.414101 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6267976f051eb6045ec6b80a00d78947d38d000021e798c1df453f426ce834e3" Oct 31 23:14:44.414219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.414121 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw" Oct 31 23:14:44.926238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:44.926145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:45.080009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:45.079961 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:45.926569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:45.919676 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:46.728014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:46.727974 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:47.089736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:47.089704 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:47.358596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:47.358462 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:47.829569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:47.827607 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:48.302350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:48.302302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:48.454394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:48.454358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:48.675063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:48.674981 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:48.864071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:48.864036 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:49.779652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:49.779622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:50.153031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.151210 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:50.189823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.189791 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt"] Oct 31 23:14:50.190952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.190925 5043 cpu_manager.go:414] "containerMap: removing container" podUID="52c58484-9578-4663-813f-ae4b5c8490f7" containerName="revoke-keys" Oct 31 23:14:50.190952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.190951 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c58484-9578-4663-813f-ae4b5c8490f7" containerName="revoke-keys" Oct 31 23:14:50.191137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.191003 5043 cpu_manager.go:414] "containerMap: removing container" podUID="28d68379-d5b2-4caf-af89-9a293fea71b7" containerName="revoke-keys" Oct 31 23:14:50.191137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.191012 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d68379-d5b2-4caf-af89-9a293fea71b7" containerName="revoke-keys" Oct 31 23:14:50.191236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.191222 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d68379-d5b2-4caf-af89-9a293fea71b7" containerName="revoke-keys" Oct 31 23:14:50.191282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.191247 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c58484-9578-4663-813f-ae4b5c8490f7" containerName="revoke-keys" Oct 31 23:14:50.199866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.199842 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.202918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.202895 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt"] Oct 31 23:14:50.205983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.205586 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:14:50.205983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.205649 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:14:50.205983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.205684 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:14:50.205983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.205760 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:14:50.306741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.306707 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.306942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.306884 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.307034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.307010 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvgb8\" (UniqueName: \"kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.409049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.408959 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.409201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.409132 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.409244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.409223 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvgb8\" (UniqueName: \"kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.409698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.409678 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.411574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.411537 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.424712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.424685 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvgb8\" (UniqueName: \"kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8\") pod \"maas-api-revoke-sub-4ec7f2fd-gw4bt\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.519394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.519343 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:50.694525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.694399 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:50.835418 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:50.835383 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93f9a34_e282_404d_b2b2_3374cdedbd6e.slice/crio-62671b1d0ec8221f82ff95edc84dc8a3acb7df10b8d61c8769105cee6209e755 WatchSource:0}: Error finding container 62671b1d0ec8221f82ff95edc84dc8a3acb7df10b8d61c8769105cee6209e755: Status 404 returned error can't find the container with id 62671b1d0ec8221f82ff95edc84dc8a3acb7df10b8d61c8769105cee6209e755 Oct 31 23:14:50.838636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:50.838606 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt"] Oct 31 23:14:51.519508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:51.519476 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:51.576136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:51.576099 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" event={"ID":"a93f9a34-e282-404d-b2b2-3374cdedbd6e","Type":"ContainerStarted","Data":"0b2d8cb318887aac6b799ba1c9322eed3426b35b4c726d09cf3a18dd26d6f7e8"} Oct 31 23:14:51.576300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:51.576153 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" event={"ID":"a93f9a34-e282-404d-b2b2-3374cdedbd6e","Type":"ContainerStarted","Data":"62671b1d0ec8221f82ff95edc84dc8a3acb7df10b8d61c8769105cee6209e755"} Oct 31 23:14:51.607091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:51.607009 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" podStartSLOduration=1.6069874629999998 podStartE2EDuration="1.606987463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:51.601792445 +0000 UTC m=+1517.831346856" watchObservedRunningTime="2026-10-31 23:14:51.606987463 +0000 UTC m=+1517.836541882" Oct 31 23:14:52.231570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.223489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:52.587474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.587438 5043 generic.go:356] "Generic (PLEG): container finished" podID="a93f9a34-e282-404d-b2b2-3374cdedbd6e" containerID="0b2d8cb318887aac6b799ba1c9322eed3426b35b4c726d09cf3a18dd26d6f7e8" exitCode=0 Oct 31 23:14:52.587692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.587528 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" event={"ID":"a93f9a34-e282-404d-b2b2-3374cdedbd6e","Type":"ContainerDied","Data":"0b2d8cb318887aac6b799ba1c9322eed3426b35b4c726d09cf3a18dd26d6f7e8"} Oct 31 23:14:52.914219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.914097 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:52.966266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.966233 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr"] Oct 31 23:14:52.971063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.971045 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:52.984321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:52.984290 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr"] Oct 31 23:14:53.078329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.078294 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.078521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.078338 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.078640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.078601 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wk46\" (UniqueName: \"kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.180701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.180621 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.180701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.180665 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.180891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.180805 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wk46\" (UniqueName: \"kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.181339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.181304 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.183126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.183104 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.193070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.193048 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wk46\" (UniqueName: \"kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46\") pod \"maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.289328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.289301 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:53.589842 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:53.589811 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0193517d_e683_4abb_bd5c_57f53ca059b7.slice/crio-e9998f6501c9bcc67b7d8e41376c45afc7d11bf6b6cd67b8d04c7e17cd9d4cd3 WatchSource:0}: Error finding container e9998f6501c9bcc67b7d8e41376c45afc7d11bf6b6cd67b8d04c7e17cd9d4cd3: Status 404 returned error can't find the container with id e9998f6501c9bcc67b7d8e41376c45afc7d11bf6b6cd67b8d04c7e17cd9d4cd3 Oct 31 23:14:53.592441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.592410 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr"] Oct 31 23:14:53.601265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.601234 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4"] Oct 31 23:14:53.618022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.610037 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.618022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.616795 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4"] Oct 31 23:14:53.693459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.693416 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.693656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.693636 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.693719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.693703 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqmj\" (UniqueName: \"kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.714846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.714816 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:53.827951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.827907 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.828311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.828291 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.837008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.836930 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.837281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.837258 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqmj\" (UniqueName: \"kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.839689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.839666 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.851006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.850981 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqmj\" (UniqueName: \"kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj\") pod \"maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.860174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.860146 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:53.940656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.940404 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca\") pod \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " Oct 31 23:14:53.940831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.940683 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca\") pod \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " Oct 31 23:14:53.940831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.940796 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8\" (UniqueName: \"kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8\") pod \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\" (UID: \"a93f9a34-e282-404d-b2b2-3374cdedbd6e\") " Oct 31 23:14:53.946529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.946226 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca" pod "a93f9a34-e282-404d-b2b2-3374cdedbd6e" (UID: "a93f9a34-e282-404d-b2b2-3374cdedbd6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:53.948285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.947988 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8" pod "a93f9a34-e282-404d-b2b2-3374cdedbd6e" (UID: "a93f9a34-e282-404d-b2b2-3374cdedbd6e"). InnerVolumeSpecName "kube-api-access-kvgb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:53.951921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.951892 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:53.954797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:53.954773 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca" pod "a93f9a34-e282-404d-b2b2-3374cdedbd6e" (UID: "a93f9a34-e282-404d-b2b2-3374cdedbd6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:54.045421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.045344 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:54.045421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.045382 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvgb8\" (UniqueName: \"kubernetes.io/projected/a93f9a34-e282-404d-b2b2-3374cdedbd6e-kube-api-access-kvgb8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:54.045421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.045399 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93f9a34-e282-404d-b2b2-3374cdedbd6e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:54.251665 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:54.251630 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04af872e_ead8_4875_81f2_563ac7a22e77.slice/crio-3ca1e78420adbcad89ee5208ce0dbbca24eaf7508c9075dacebf9816cc83d87d WatchSource:0}: Error finding container 3ca1e78420adbcad89ee5208ce0dbbca24eaf7508c9075dacebf9816cc83d87d: Status 404 returned error can't find the container with id 3ca1e78420adbcad89ee5208ce0dbbca24eaf7508c9075dacebf9816cc83d87d Oct 31 23:14:54.255194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.255166 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4"] Oct 31 23:14:54.509319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.509286 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:54.613495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.613466 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" Oct 31 23:14:54.613495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.613479 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt" event={"ID":"a93f9a34-e282-404d-b2b2-3374cdedbd6e","Type":"ContainerDied","Data":"62671b1d0ec8221f82ff95edc84dc8a3acb7df10b8d61c8769105cee6209e755"} Oct 31 23:14:54.613781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.613531 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62671b1d0ec8221f82ff95edc84dc8a3acb7df10b8d61c8769105cee6209e755" Oct 31 23:14:54.621601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.618655 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" event={"ID":"0193517d-e683-4abb-bd5c-57f53ca059b7","Type":"ContainerStarted","Data":"95ad86dc72a7c50f2df2720af645ae7eaa169d811aad386bf2a1a41632c26e53"} Oct 31 23:14:54.621601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.618743 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" event={"ID":"0193517d-e683-4abb-bd5c-57f53ca059b7","Type":"ContainerStarted","Data":"e9998f6501c9bcc67b7d8e41376c45afc7d11bf6b6cd67b8d04c7e17cd9d4cd3"} Oct 31 23:14:54.621601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.620160 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" event={"ID":"04af872e-ead8-4875-81f2-563ac7a22e77","Type":"ContainerStarted","Data":"3ca1e78420adbcad89ee5208ce0dbbca24eaf7508c9075dacebf9816cc83d87d"} Oct 31 23:14:54.646248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:54.646196 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" podStartSLOduration=2.646179604 podStartE2EDuration="2.646179604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:14:54.6372737 +0000 UTC m=+1520.866828111" watchObservedRunningTime="2026-10-31 23:14:54.646179604 +0000 UTC m=+1520.875734014" Oct 31 23:14:55.351598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.351174 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:55.498617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.498583 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:55.634991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.634900 5043 generic.go:356] "Generic (PLEG): container finished" podID="0193517d-e683-4abb-bd5c-57f53ca059b7" containerID="95ad86dc72a7c50f2df2720af645ae7eaa169d811aad386bf2a1a41632c26e53" exitCode=0 Oct 31 23:14:55.635357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.634984 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" event={"ID":"0193517d-e683-4abb-bd5c-57f53ca059b7","Type":"ContainerDied","Data":"95ad86dc72a7c50f2df2720af645ae7eaa169d811aad386bf2a1a41632c26e53"} Oct 31 23:14:55.636956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.636928 5043 generic.go:356] "Generic (PLEG): container finished" podID="04af872e-ead8-4875-81f2-563ac7a22e77" containerID="04c905332395c12b004e61419e22a2cd7d145302a9ac34e17ff50b5eab4c4223" exitCode=0 Oct 31 23:14:55.637076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.637007 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" event={"ID":"04af872e-ead8-4875-81f2-563ac7a22e77","Type":"ContainerDied","Data":"04c905332395c12b004e61419e22a2cd7d145302a9ac34e17ff50b5eab4c4223"} Oct 31 23:14:55.787161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:55.787125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:56.247035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.246997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:56.882880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.882854 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:56.888025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.888002 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:56.912148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.910283 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:56.966569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.948859 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca\") pod \"04af872e-ead8-4875-81f2-563ac7a22e77\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " Oct 31 23:14:56.966569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.949032 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca\") pod \"04af872e-ead8-4875-81f2-563ac7a22e77\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " Oct 31 23:14:56.966569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.949175 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj\" (UniqueName: \"kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj\") pod \"04af872e-ead8-4875-81f2-563ac7a22e77\" (UID: \"04af872e-ead8-4875-81f2-563ac7a22e77\") " Oct 31 23:14:56.980584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.974267 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca" pod "04af872e-ead8-4875-81f2-563ac7a22e77" (UID: "04af872e-ead8-4875-81f2-563ac7a22e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:56.980584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:56.976078 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca" pod "04af872e-ead8-4875-81f2-563ac7a22e77" (UID: "04af872e-ead8-4875-81f2-563ac7a22e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:57.023647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.018872 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj" pod "04af872e-ead8-4875-81f2-563ac7a22e77" (UID: "04af872e-ead8-4875-81f2-563ac7a22e77"). InnerVolumeSpecName "kube-api-access-khqmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:57.066482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.066449 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46\" (UniqueName: \"kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46\") pod \"0193517d-e683-4abb-bd5c-57f53ca059b7\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " Oct 31 23:14:57.066712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.066516 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca\") pod \"0193517d-e683-4abb-bd5c-57f53ca059b7\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " Oct 31 23:14:57.066785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.066752 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca\") pod \"0193517d-e683-4abb-bd5c-57f53ca059b7\" (UID: \"0193517d-e683-4abb-bd5c-57f53ca059b7\") " Oct 31 23:14:57.068083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.068057 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca" pod "0193517d-e683-4abb-bd5c-57f53ca059b7" (UID: "0193517d-e683-4abb-bd5c-57f53ca059b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:14:57.069601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.068847 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:57.069601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.068874 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:57.069601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.068888 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqmj\" (UniqueName: \"kubernetes.io/projected/04af872e-ead8-4875-81f2-563ac7a22e77-kube-api-access-khqmj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:57.069601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.068902 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04af872e-ead8-4875-81f2-563ac7a22e77-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:57.069856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.069799 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46" pod "0193517d-e683-4abb-bd5c-57f53ca059b7" (UID: "0193517d-e683-4abb-bd5c-57f53ca059b7"). InnerVolumeSpecName "kube-api-access-2wk46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:14:57.070664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.070638 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca" pod "0193517d-e683-4abb-bd5c-57f53ca059b7" (UID: "0193517d-e683-4abb-bd5c-57f53ca059b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:14:57.170278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.170194 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0193517d-e683-4abb-bd5c-57f53ca059b7-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:57.170278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.170228 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wk46\" (UniqueName: \"kubernetes.io/projected/0193517d-e683-4abb-bd5c-57f53ca059b7-kube-api-access-2wk46\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:14:57.374766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.374734 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl"] Oct 31 23:14:57.375442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375427 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a93f9a34-e282-404d-b2b2-3374cdedbd6e" containerName="revoke-keys" Oct 31 23:14:57.375497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375442 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93f9a34-e282-404d-b2b2-3374cdedbd6e" containerName="revoke-keys" Oct 31 23:14:57.375497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375463 5043 cpu_manager.go:414] "containerMap: removing container" podUID="04af872e-ead8-4875-81f2-563ac7a22e77" containerName="revoke-keys" Oct 31 23:14:57.375497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375468 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04af872e-ead8-4875-81f2-563ac7a22e77" containerName="revoke-keys" Oct 31 23:14:57.375497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375487 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0193517d-e683-4abb-bd5c-57f53ca059b7" containerName="revoke-keys" Oct 31 23:14:57.375497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375492 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0193517d-e683-4abb-bd5c-57f53ca059b7" containerName="revoke-keys" Oct 31 23:14:57.375657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375607 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0193517d-e683-4abb-bd5c-57f53ca059b7" containerName="revoke-keys" Oct 31 23:14:57.375657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375624 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93f9a34-e282-404d-b2b2-3374cdedbd6e" containerName="revoke-keys" Oct 31 23:14:57.375657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.375632 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="04af872e-ead8-4875-81f2-563ac7a22e77" containerName="revoke-keys" Oct 31 23:14:57.380304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.380283 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.392193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.392164 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 23:14:57.393037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.392249 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 23:14:57.393037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.392279 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 23:14:57.393037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.392363 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v7rzj\"" Oct 31 23:14:57.393219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.393137 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl"] Oct 31 23:14:57.474595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.474468 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.474595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.474503 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.474844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.474690 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.474916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.474855 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.474916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.474878 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k5c\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.474998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.474968 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.475127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.475108 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577092 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577318 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577347 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577421 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577606 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577645 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k5c\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.577749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.577698 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.578156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.578132 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.580936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.580907 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.581090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.580913 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.584097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.583170 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.598875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.598024 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.598875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.598034 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.602644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.602617 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k5c\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xbgvl\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:57.660597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.660532 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" Oct 31 23:14:57.661209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.661176 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr" event={"ID":"0193517d-e683-4abb-bd5c-57f53ca059b7","Type":"ContainerDied","Data":"e9998f6501c9bcc67b7d8e41376c45afc7d11bf6b6cd67b8d04c7e17cd9d4cd3"} Oct 31 23:14:57.661363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.661230 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9998f6501c9bcc67b7d8e41376c45afc7d11bf6b6cd67b8d04c7e17cd9d4cd3" Oct 31 23:14:57.663092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.663063 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" event={"ID":"04af872e-ead8-4875-81f2-563ac7a22e77","Type":"ContainerDied","Data":"3ca1e78420adbcad89ee5208ce0dbbca24eaf7508c9075dacebf9816cc83d87d"} Oct 31 23:14:57.663191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.663109 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ca1e78420adbcad89ee5208ce0dbbca24eaf7508c9075dacebf9816cc83d87d" Oct 31 23:14:57.663191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.663166 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4" Oct 31 23:14:57.703281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:57.703258 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:14:58.005675 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:14:58.005649 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ae1e98_0054_4490_9a79_56b82a6428a3.slice/crio-c12b2ef948b1e207f3213a10f5423891ab1578d917115c9b6db3c3af1fba9fb2 WatchSource:0}: Error finding container c12b2ef948b1e207f3213a10f5423891ab1578d917115c9b6db3c3af1fba9fb2: Status 404 returned error can't find the container with id c12b2ef948b1e207f3213a10f5423891ab1578d917115c9b6db3c3af1fba9fb2 Oct 31 23:14:58.010171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:58.010148 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl"] Oct 31 23:14:58.300445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:58.300411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:58.674381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:58.674289 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" event={"ID":"18ae1e98-0054-4490-9a79-56b82a6428a3","Type":"ContainerStarted","Data":"c12b2ef948b1e207f3213a10f5423891ab1578d917115c9b6db3c3af1fba9fb2"} Oct 31 23:14:58.701902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:58.701874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:14:59.337475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:14:59.337431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:00.030039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.029974 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:00.184437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.184401 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6"] Oct 31 23:15:00.192332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.192305 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.208775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.208748 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:15:00.208775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.208749 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:15:00.208949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.208805 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:15:00.210116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.209933 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:15:00.211689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.211666 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6"] Oct 31 23:15:00.221829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.221801 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k"] Oct 31 23:15:00.229601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.229521 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.232162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.232136 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjk6f\" (UniqueName: \"kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.232418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.232398 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.232702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.232680 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.233028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.233010 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4"] Oct 31 23:15:00.242744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.242724 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.260648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.260617 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr"] Oct 31 23:15:00.283786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.281648 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.283786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.281839 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn"] Oct 31 23:15:00.297954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.297923 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k"] Oct 31 23:15:00.298077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.298062 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.306809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.306784 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4"] Oct 31 23:15:00.318131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.318106 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr"] Oct 31 23:15:00.335490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.335456 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.335650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.335575 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjk6f\" (UniqueName: \"kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.335717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.335704 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.335780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.335744 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rppg\" (UniqueName: \"kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg\") pod \"maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.336296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.336270 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glw8\" (UniqueName: \"kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8\") pod \"maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.336427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.336408 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.336930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.336904 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.337708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.337680 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.341122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.341099 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.352499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.351828 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn"] Oct 31 23:15:00.369771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.368483 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln"] Oct 31 23:15:00.379019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.375104 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.379019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.375267 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjk6f\" (UniqueName: \"kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f\") pod \"maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.380716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.380498 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj"] Oct 31 23:15:00.404640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.404578 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.412594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.410030 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln"] Oct 31 23:15:00.424612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.419918 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj"] Oct 31 23:15:00.438464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.438424 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.438707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.438514 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4sx\" (UniqueName: \"kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx\") pod \"maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.438707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.438607 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rppg\" (UniqueName: \"kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg\") pod \"maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.438707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.438661 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcf6\" (UniqueName: \"kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6\") pod \"maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.439023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.438998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6zj\" (UniqueName: \"kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj\") pod \"maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.439115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.439086 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.439204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.439172 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glw8\" (UniqueName: \"kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8\") pod \"maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.439272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.439226 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.439331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.439311 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891475-h5sln\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.439398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.439382 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.440343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.440320 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.440495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.440469 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.440621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.440601 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6mh\" (UniqueName: \"kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh\") pod \"maas-api-key-cleanup-29891475-h5sln\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.441818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.441798 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.465290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.455291 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glw8\" (UniqueName: \"kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8\") pod \"maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.477730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.470783 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rppg\" (UniqueName: \"kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg\") pod \"maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.521505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.521469 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:00.543816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.543788 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6mh\" (UniqueName: \"kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh\") pod \"maas-api-key-cleanup-29891475-h5sln\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.544102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544074 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.544441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544417 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4sx\" (UniqueName: \"kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx\") pod \"maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.544506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544466 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcf6\" (UniqueName: \"kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6\") pod \"maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.544572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544502 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6zj\" (UniqueName: \"kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj\") pod \"maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.544633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544598 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.544745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544728 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891475-h5sln\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.544800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.544778 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.545443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.545418 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.545712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.545691 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.546387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.546365 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891475-h5sln\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.546455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.546409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.554969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.554904 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:00.558427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.558388 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6mh\" (UniqueName: \"kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh\") pod \"maas-api-key-cleanup-29891475-h5sln\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.558877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.558855 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcf6\" (UniqueName: \"kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6\") pod \"maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.560904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.560883 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4sx\" (UniqueName: \"kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx\") pod \"maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.564513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.564493 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6zj\" (UniqueName: \"kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj\") pod \"maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.570402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.570358 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:00.607832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.607518 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:00.629094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.621704 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:00.691672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.691528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:00.747495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.747087 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:00.750649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.750563 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:00.848135 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:00.848095 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610be94b_a435_4d5c_a1f9_71b5fc66b590.slice/crio-3f956b01d955113712e050069457ad9e2ad16011958658afb136136310f25759 WatchSource:0}: Error finding container 3f956b01d955113712e050069457ad9e2ad16011958658afb136136310f25759: Status 404 returned error can't find the container with id 3f956b01d955113712e050069457ad9e2ad16011958658afb136136310f25759 Oct 31 23:15:00.852101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:00.851059 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6"] Oct 31 23:15:01.070982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.070825 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:01.139647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.139583 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k"] Oct 31 23:15:01.141954 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:01.141932 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb819fdb2_a681_41df_9020_be1f3f0f1e44.slice/crio-3f5dc6959e4009241a04df7e5c3e3ede74fc2cb1d96cbf848ff9bd209a3851df WatchSource:0}: Error finding container 3f5dc6959e4009241a04df7e5c3e3ede74fc2cb1d96cbf848ff9bd209a3851df: Status 404 returned error can't find the container with id 3f5dc6959e4009241a04df7e5c3e3ede74fc2cb1d96cbf848ff9bd209a3851df Oct 31 23:15:01.148713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.148690 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4"] Oct 31 23:15:01.291081 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:01.291041 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003fa55a_443d_4e54_8842_f426d9d46ad5.slice/crio-1717922d27630502b9535d2586dfd7944e0e54a140d0ec5217f615f9d10a0f81 WatchSource:0}: Error finding container 1717922d27630502b9535d2586dfd7944e0e54a140d0ec5217f615f9d10a0f81: Status 404 returned error can't find the container with id 1717922d27630502b9535d2586dfd7944e0e54a140d0ec5217f615f9d10a0f81 Oct 31 23:15:01.296876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.296852 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln"] Oct 31 23:15:01.358819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.358786 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj"] Oct 31 23:15:01.373658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.373628 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr"] Oct 31 23:15:01.383953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.383930 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn"] Oct 31 23:15:01.442877 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:01.442848 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f788bf6_0c66_4d64_9495_b46fb49037d7.slice/crio-377db94d1316d3f4ec88e9c641edf1e25aa2e053df660cb6e8326c750779ba00 WatchSource:0}: Error finding container 377db94d1316d3f4ec88e9c641edf1e25aa2e053df660cb6e8326c750779ba00: Status 404 returned error can't find the container with id 377db94d1316d3f4ec88e9c641edf1e25aa2e053df660cb6e8326c750779ba00 Oct 31 23:15:01.747525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.747434 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" event={"ID":"b819fdb2-a681-41df-9020-be1f3f0f1e44","Type":"ContainerStarted","Data":"3f5dc6959e4009241a04df7e5c3e3ede74fc2cb1d96cbf848ff9bd209a3851df"} Oct 31 23:15:01.749420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.749341 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerStarted","Data":"231d195df06e3ff056c63a2be126657cb7f31abf829638f71b1914a18194185d"} Oct 31 23:15:01.751494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.751421 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerStarted","Data":"377db94d1316d3f4ec88e9c641edf1e25aa2e053df660cb6e8326c750779ba00"} Oct 31 23:15:01.753123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.753059 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" event={"ID":"e96d5f04-c004-4d54-836a-f040ff8b99ca","Type":"ContainerStarted","Data":"fa6548dc76db8f2b0dd258f2283b14dfdece26ccd32cdb57429833173f3de06f"} Oct 31 23:15:01.755876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.755042 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" event={"ID":"610be94b-a435-4d5c-a1f9-71b5fc66b590","Type":"ContainerStarted","Data":"948f0c81bdb8dde619333b40e0c0b1ec042451eb4deab727cb70e6c10f12d08b"} Oct 31 23:15:01.755876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.755082 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" event={"ID":"610be94b-a435-4d5c-a1f9-71b5fc66b590","Type":"ContainerStarted","Data":"3f956b01d955113712e050069457ad9e2ad16011958658afb136136310f25759"} Oct 31 23:15:01.757175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.757131 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" event={"ID":"003fa55a-443d-4e54-8842-f426d9d46ad5","Type":"ContainerStarted","Data":"1717922d27630502b9535d2586dfd7944e0e54a140d0ec5217f615f9d10a0f81"} Oct 31 23:15:01.758800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.758780 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerStarted","Data":"afcfe26073976a51a98181dacd8ce8472ff2541be0dd2be8c398d073ce7b2d6b"} Oct 31 23:15:01.780102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.780041 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" podStartSLOduration=1.780026156 podStartE2EDuration="1.780026156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:01.773572793 +0000 UTC m=+1528.003127203" watchObservedRunningTime="2026-10-31 23:15:01.780026156 +0000 UTC m=+1528.009580568" Oct 31 23:15:01.873821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:01.873782 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:02.122265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.122151 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:02.258108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.258029 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:02.552213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.552125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:02.772798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.772770 5043 generic.go:356] "Generic (PLEG): container finished" podID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerID="c8526c384d910126256f586bece0e2ec72674ad614fcbc49d56505f2d104268c" exitCode=60 Oct 31 23:15:02.772914 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.772852 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" event={"ID":"b819fdb2-a681-41df-9020-be1f3f0f1e44","Type":"ContainerDied","Data":"c8526c384d910126256f586bece0e2ec72674ad614fcbc49d56505f2d104268c"} Oct 31 23:15:02.773150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.773136 5043 scope.go:122] "RemoveContainer" containerID="c8526c384d910126256f586bece0e2ec72674ad614fcbc49d56505f2d104268c" Oct 31 23:15:02.774692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.774664 5043 generic.go:356] "Generic (PLEG): container finished" podID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerID="4196fbbddf53bc1422646fd37406154e96a1d80088f60e35ba51e1f9c01a1605" exitCode=60 Oct 31 23:15:02.774800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.774750 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerDied","Data":"4196fbbddf53bc1422646fd37406154e96a1d80088f60e35ba51e1f9c01a1605"} Oct 31 23:15:02.774969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.774953 5043 scope.go:122] "RemoveContainer" containerID="4196fbbddf53bc1422646fd37406154e96a1d80088f60e35ba51e1f9c01a1605" Oct 31 23:15:02.776685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.776658 5043 generic.go:356] "Generic (PLEG): container finished" podID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerID="3b279ff96315dd73b19e01e7d068ce248405369ccac220666ae0d2abd060c4d9" exitCode=60 Oct 31 23:15:02.776775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.776723 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerDied","Data":"3b279ff96315dd73b19e01e7d068ce248405369ccac220666ae0d2abd060c4d9"} Oct 31 23:15:02.776980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.776966 5043 scope.go:122] "RemoveContainer" containerID="3b279ff96315dd73b19e01e7d068ce248405369ccac220666ae0d2abd060c4d9" Oct 31 23:15:02.778482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.778461 5043 generic.go:356] "Generic (PLEG): container finished" podID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerID="c21bfa21c48cad692e1f8119ac55591480bb895f2f79801ad118c51ac7752184" exitCode=60 Oct 31 23:15:02.778613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.778554 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" event={"ID":"e96d5f04-c004-4d54-836a-f040ff8b99ca","Type":"ContainerDied","Data":"c21bfa21c48cad692e1f8119ac55591480bb895f2f79801ad118c51ac7752184"} Oct 31 23:15:02.778766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.778751 5043 scope.go:122] "RemoveContainer" containerID="c21bfa21c48cad692e1f8119ac55591480bb895f2f79801ad118c51ac7752184" Oct 31 23:15:02.780156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.780139 5043 generic.go:356] "Generic (PLEG): container finished" podID="610be94b-a435-4d5c-a1f9-71b5fc66b590" containerID="948f0c81bdb8dde619333b40e0c0b1ec042451eb4deab727cb70e6c10f12d08b" exitCode=0 Oct 31 23:15:02.780252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.780203 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" event={"ID":"610be94b-a435-4d5c-a1f9-71b5fc66b590","Type":"ContainerDied","Data":"948f0c81bdb8dde619333b40e0c0b1ec042451eb4deab727cb70e6c10f12d08b"} Oct 31 23:15:02.781742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.781724 5043 generic.go:356] "Generic (PLEG): container finished" podID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerID="1f604b01843269ae112bb653a7b9b94e4975c1c80292d28a85336b9309d973c9" exitCode=60 Oct 31 23:15:02.781816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.781798 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" event={"ID":"003fa55a-443d-4e54-8842-f426d9d46ad5","Type":"ContainerDied","Data":"1f604b01843269ae112bb653a7b9b94e4975c1c80292d28a85336b9309d973c9"} Oct 31 23:15:02.781989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.781972 5043 scope.go:122] "RemoveContainer" containerID="1f604b01843269ae112bb653a7b9b94e4975c1c80292d28a85336b9309d973c9" Oct 31 23:15:02.783458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.783439 5043 generic.go:356] "Generic (PLEG): container finished" podID="99c12441-fdb1-4920-aeef-3a351971a5df" containerID="a1e77bb33085b06d2aa7d2d855ab098979e095930ff5e7fd0a906988479538f9" exitCode=60 Oct 31 23:15:02.783556 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.783522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerDied","Data":"a1e77bb33085b06d2aa7d2d855ab098979e095930ff5e7fd0a906988479538f9"} Oct 31 23:15:02.783715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.783703 5043 scope.go:122] "RemoveContainer" containerID="a1e77bb33085b06d2aa7d2d855ab098979e095930ff5e7fd0a906988479538f9" Oct 31 23:15:02.890384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.890353 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:02.890585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:02.890527 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:03.147290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.147255 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4"] Oct 31 23:15:03.240105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.240067 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:03.403523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.398034 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:03.790221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.789683 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7"] Oct 31 23:15:03.846789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.846657 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:03.851099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.850337 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd"] Oct 31 23:15:03.861144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.860761 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerStarted","Data":"ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2"} Oct 31 23:15:03.865739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:03.864905 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" event={"ID":"e96d5f04-c004-4d54-836a-f040ff8b99ca","Type":"ContainerStarted","Data":"812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b"} Oct 31 23:15:04.077506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.077472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:04.405290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.399083 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" podStartSLOduration=4.399065335 podStartE2EDuration="4.399065335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:03.921834528 +0000 UTC m=+1530.151388939" watchObservedRunningTime="2026-10-31 23:15:04.399065335 +0000 UTC m=+1530.628619787" Oct 31 23:15:04.435424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.435387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:04.457294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.457181 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" event={"ID":"003fa55a-443d-4e54-8842-f426d9d46ad5","Type":"ContainerStarted","Data":"ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39"} Oct 31 23:15:04.459899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.458973 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerStarted","Data":"4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639"} Oct 31 23:15:04.463334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.462269 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" podStartSLOduration=4.4622522 podStartE2EDuration="4.4622522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:03.948397076 +0000 UTC m=+1530.177951514" watchObservedRunningTime="2026-10-31 23:15:04.4622522 +0000 UTC m=+1530.691806611" Oct 31 23:15:04.464677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.464653 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" event={"ID":"b819fdb2-a681-41df-9020-be1f3f0f1e44","Type":"ContainerStarted","Data":"9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6"} Oct 31 23:15:04.464906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.464870 5043 scope.go:122] "RemoveContainer" containerID="4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639" Oct 31 23:15:04.465725 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:04.465357 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr_odh-ai-gateway-infra(99c12441-fdb1-4920-aeef-3a351971a5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" Oct 31 23:15:04.466852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.466832 5043 scope.go:122] "RemoveContainer" containerID="9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6" Oct 31 23:15:04.467572 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:04.467428 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4_odh-ai-gateway-infra(b819fdb2-a681-41df-9020-be1f3f0f1e44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" Oct 31 23:15:04.480181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.469584 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" event={"ID":"18ae1e98-0054-4490-9a79-56b82a6428a3","Type":"ContainerStarted","Data":"0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c"} Oct 31 23:15:04.480345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.480327 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerStarted","Data":"4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723"} Oct 31 23:15:04.480459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.480447 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:15:04.480568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.477639 5043 scope.go:122] "RemoveContainer" containerID="4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723" Oct 31 23:15:04.481028 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:04.481003 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn_odh-ai-gateway-infra(fa0af7e9-5bed-41ab-9cf1-155215ec20f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" Oct 31 23:15:04.481145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.474893 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" podStartSLOduration=4.474876474 podStartE2EDuration="4.474876474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:04.30277058 +0000 UTC m=+1530.532324990" watchObservedRunningTime="2026-10-31 23:15:04.474876474 +0000 UTC m=+1530.704430884" Oct 31 23:15:04.482398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.482310 5043 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xbgvl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:15:04.482398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.482348 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:15:04.588921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.588512 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" podStartSLOduration=2.7070905119999997 podStartE2EDuration="7.588493379s" totalImagesPullingTime="4.881402867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:04.573272095 +0000 UTC m=+1530.802826507" watchObservedRunningTime="2026-10-31 23:15:04.588493379 +0000 UTC m=+1530.818047784" Oct 31 23:15:04.780710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.780682 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:04.785450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.785429 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca\") pod \"610be94b-a435-4d5c-a1f9-71b5fc66b590\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " Oct 31 23:15:04.785522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.785505 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f\" (UniqueName: \"kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f\") pod \"610be94b-a435-4d5c-a1f9-71b5fc66b590\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " Oct 31 23:15:04.785866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.785838 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca\") pod \"610be94b-a435-4d5c-a1f9-71b5fc66b590\" (UID: \"610be94b-a435-4d5c-a1f9-71b5fc66b590\") " Oct 31 23:15:04.786133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.786098 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca" pod "610be94b-a435-4d5c-a1f9-71b5fc66b590" (UID: "610be94b-a435-4d5c-a1f9-71b5fc66b590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:04.786615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.786590 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:04.788886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.788856 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca" pod "610be94b-a435-4d5c-a1f9-71b5fc66b590" (UID: "610be94b-a435-4d5c-a1f9-71b5fc66b590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:04.788964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.788944 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f" pod "610be94b-a435-4d5c-a1f9-71b5fc66b590" (UID: "610be94b-a435-4d5c-a1f9-71b5fc66b590"). InnerVolumeSpecName "kube-api-access-wjk6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:04.888103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.888018 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610be94b-a435-4d5c-a1f9-71b5fc66b590-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:04.888103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:04.888060 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjk6f\" (UniqueName: \"kubernetes.io/projected/610be94b-a435-4d5c-a1f9-71b5fc66b590-kube-api-access-wjk6f\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:05.111110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.111069 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:05.494390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.494350 5043 generic.go:356] "Generic (PLEG): container finished" podID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerID="9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6" exitCode=60 Oct 31 23:15:05.494632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.494434 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" event={"ID":"b819fdb2-a681-41df-9020-be1f3f0f1e44","Type":"ContainerDied","Data":"9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6"} Oct 31 23:15:05.494632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.494535 5043 scope.go:122] "RemoveContainer" containerID="c8526c384d910126256f586bece0e2ec72674ad614fcbc49d56505f2d104268c" Oct 31 23:15:05.494876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.494859 5043 scope.go:122] "RemoveContainer" containerID="9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6" Oct 31 23:15:05.495346 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:05.495314 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4_odh-ai-gateway-infra(b819fdb2-a681-41df-9020-be1f3f0f1e44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" Oct 31 23:15:05.497072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.497050 5043 generic.go:356] "Generic (PLEG): container finished" podID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerID="4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723" exitCode=60 Oct 31 23:15:05.497161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.497146 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerDied","Data":"4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723"} Oct 31 23:15:05.497671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.497654 5043 scope.go:122] "RemoveContainer" containerID="4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723" Oct 31 23:15:05.498088 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:05.498065 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn_odh-ai-gateway-infra(fa0af7e9-5bed-41ab-9cf1-155215ec20f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" Oct 31 23:15:05.500232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.500208 5043 generic.go:356] "Generic (PLEG): container finished" podID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerID="ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2" exitCode=60 Oct 31 23:15:05.500316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.500285 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerDied","Data":"ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2"} Oct 31 23:15:05.500777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.500754 5043 scope.go:122] "RemoveContainer" containerID="ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2" Oct 31 23:15:05.501188 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:05.501162 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj_odh-ai-gateway-infra(3f788bf6-0c66-4d64-9495-b46fb49037d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" Oct 31 23:15:05.502903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.502879 5043 generic.go:356] "Generic (PLEG): container finished" podID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerID="812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b" exitCode=60 Oct 31 23:15:05.503007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.502954 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" event={"ID":"e96d5f04-c004-4d54-836a-f040ff8b99ca","Type":"ContainerDied","Data":"812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b"} Oct 31 23:15:05.503214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.503195 5043 scope.go:122] "RemoveContainer" containerID="812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b" Oct 31 23:15:05.503507 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:05.503480 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k_odh-ai-gateway-infra(e96d5f04-c004-4d54-836a-f040ff8b99ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" Oct 31 23:15:05.504611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.504591 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" Oct 31 23:15:05.504611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.504602 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6" event={"ID":"610be94b-a435-4d5c-a1f9-71b5fc66b590","Type":"ContainerDied","Data":"3f956b01d955113712e050069457ad9e2ad16011958658afb136136310f25759"} Oct 31 23:15:05.504786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.504639 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f956b01d955113712e050069457ad9e2ad16011958658afb136136310f25759" Oct 31 23:15:05.507906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.507885 5043 generic.go:356] "Generic (PLEG): container finished" podID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerID="ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39" exitCode=60 Oct 31 23:15:05.508039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.508024 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" event={"ID":"003fa55a-443d-4e54-8842-f426d9d46ad5","Type":"ContainerDied","Data":"ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39"} Oct 31 23:15:05.508293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.508271 5043 scope.go:122] "RemoveContainer" containerID="ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39" Oct 31 23:15:05.508683 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:05.508657 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891475-h5sln_odh-ai-gateway-infra(003fa55a-443d-4e54-8842-f426d9d46ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" Oct 31 23:15:05.511665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.510161 5043 generic.go:356] "Generic (PLEG): container finished" podID="99c12441-fdb1-4920-aeef-3a351971a5df" containerID="4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639" exitCode=60 Oct 31 23:15:05.511665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.510320 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerDied","Data":"4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639"} Oct 31 23:15:05.511665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.510585 5043 scope.go:122] "RemoveContainer" containerID="4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639" Oct 31 23:15:05.511665 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:05.510844 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr_odh-ai-gateway-infra(99c12441-fdb1-4920-aeef-3a351971a5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" Oct 31 23:15:05.512794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.512778 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:15:05.515423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.515406 5043 scope.go:122] "RemoveContainer" containerID="4196fbbddf53bc1422646fd37406154e96a1d80088f60e35ba51e1f9c01a1605" Oct 31 23:15:05.534146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.534123 5043 scope.go:122] "RemoveContainer" containerID="3b279ff96315dd73b19e01e7d068ce248405369ccac220666ae0d2abd060c4d9" Oct 31 23:15:05.554203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.554178 5043 scope.go:122] "RemoveContainer" containerID="c21bfa21c48cad692e1f8119ac55591480bb895f2f79801ad118c51ac7752184" Oct 31 23:15:05.584178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.584156 5043 scope.go:122] "RemoveContainer" containerID="1f604b01843269ae112bb653a7b9b94e4975c1c80292d28a85336b9309d973c9" Oct 31 23:15:05.635220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.635199 5043 scope.go:122] "RemoveContainer" containerID="a1e77bb33085b06d2aa7d2d855ab098979e095930ff5e7fd0a906988479538f9" Oct 31 23:15:05.771651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.771568 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:05.951646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.951574 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:05.995153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:05.995121 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4"] Oct 31 23:15:06.003184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.003153 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b2c6553b-xj5g4"] Oct 31 23:15:06.092611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.092122 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:06.402627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.394319 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:06.732779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.700666 5043 scope.go:122] "RemoveContainer" containerID="ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39" Oct 31 23:15:06.767678 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:06.737367 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891475-h5sln_odh-ai-gateway-infra(003fa55a-443d-4e54-8842-f426d9d46ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" Oct 31 23:15:06.922568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.922086 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.995919 5043 scope.go:122] "RemoveContainer" containerID="4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:06.996691 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr_odh-ai-gateway-infra(99c12441-fdb1-4920-aeef-3a351971a5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.996981 5043 scope.go:122] "RemoveContainer" containerID="9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:06.997477 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4_odh-ai-gateway-infra(b819fdb2-a681-41df-9020-be1f3f0f1e44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:06.998031 5043 scope.go:122] "RemoveContainer" containerID="4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:06.998475 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn_odh-ai-gateway-infra(fa0af7e9-5bed-41ab-9cf1-155215ec20f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.005531 5043 scope.go:122] "RemoveContainer" containerID="ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2" Oct 31 23:15:07.009566 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:07.005984 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj_odh-ai-gateway-infra(3f788bf6-0c66-4d64-9495-b46fb49037d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" Oct 31 23:15:07.010383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.009868 5043 scope.go:122] "RemoveContainer" containerID="812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b" Oct 31 23:15:07.015631 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:07.010980 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k_odh-ai-gateway-infra(e96d5f04-c004-4d54-836a-f040ff8b99ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" Oct 31 23:15:07.061567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.017406 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv"] Oct 31 23:15:07.061567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.017740 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" podUID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" containerName="istio-proxy" containerID="cri-o://d5b88374f2d00a8bdda3dc21ec7d90765349a8c790b50f263735665dd191ecd7" gracePeriod=30 Oct 31 23:15:07.240149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.196260 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:07.418622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.410918 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr"] Oct 31 23:15:07.466566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.434840 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:07.667330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.667289 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:07.861571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.842655 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:07.904332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.903086 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:07.905607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.904603 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:07.907277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.905818 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:07.911590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.907418 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded" Oct 31 23:15:07.911590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:07.907473 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:08.034430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.034392 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04af872e-ead8-4875-81f2-563ac7a22e77" path="/var/lib/kubelet/pods/04af872e-ead8-4875-81f2-563ac7a22e77/volumes" Oct 31 23:15:08.052010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.051959 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed6117ae-9vzrr"] Oct 31 23:15:08.103231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.103187 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.132244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.113032 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:08.132956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.132911 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.134340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.134310 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:08.167936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.166627 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" podUID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" containerName="payload-pre-processing" containerID="cri-o://df0c471c5ce9e9da0e829ab95bf304e1556f8e13f5011d01bc8a85879b1ef832" gracePeriod=25 Oct 31 23:15:08.198271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.197643 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.344651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.332717 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:08.372966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.371883 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:15:08.555605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.554527 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:15:08.588565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.563071 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:08.674583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.667944 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:08.756617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.753711 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:08.793590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.790995 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" podUID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" containerName="payload-processing" containerID="cri-o://407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9" gracePeriod=25 Oct 31 23:15:08.843749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.843653 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:08.893571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.853761 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" podUID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" containerName="payload-pre-processing" containerID="cri-o://44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50" gracePeriod=25 Oct 31 23:15:08.923884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.892662 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:15:08.948568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.920773 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:09.086955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.920807 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:09.086955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.075785 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:15:09.086955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.075849 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:15:09.112622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.920827 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:09.112622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.102377 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:15:09.112622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:08.920846 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:09.112622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.103504 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:15:09.112622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.104050 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:09.112622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.104438 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:15:09.138568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.137753 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.171986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.167884 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:15:09.171986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.168250 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:09.171986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.168724 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:15:09.242056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.242008 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.242313 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.242281 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:09.242777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.242756 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:15:09.242874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.242818 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:15:09.553702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.553655 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:09.664006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.663919 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:15:09.777574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.756094 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb"] Oct 31 23:15:09.777574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.757460 5043 cpu_manager.go:414] "containerMap: removing container" podUID="610be94b-a435-4d5c-a1f9-71b5fc66b590" containerName="revoke-keys" Oct 31 23:15:09.777574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.757485 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610be94b-a435-4d5c-a1f9-71b5fc66b590" containerName="revoke-keys" Oct 31 23:15:09.777574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.757732 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="610be94b-a435-4d5c-a1f9-71b5fc66b590" containerName="revoke-keys" Oct 31 23:15:09.802569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.784243 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.802569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.785215 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:09.802569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.785712 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:15:09.802569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.802157 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.802569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.802204 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.840635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.812961 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-907d182f-openshift-default-dockercfg-vdf9p\"" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841317 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841362 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841449 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841481 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841529 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841842 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:09.844576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.841880 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.957692 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb"] Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.957779 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959137 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959180 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959239 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959263 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959321 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959352 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959393 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959619 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.959652 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.961065 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.987721 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.988476 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.011628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:09.989170 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.136472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.090573 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.142803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.142724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:15:10.143065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.142984 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.191412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.143308 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.191412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.143662 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:10.191412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.143830 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:10.191412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.143987 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:10.191412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.153085 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.239824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.194121 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.239824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.194968 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 31 23:15:10.240881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.240765 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:10.302617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.277516 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.324444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.299072 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:10.868687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:10.842336 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.189076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.102361 5043 generic.go:356] "Generic (PLEG): container finished" podID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" containerID="df0c471c5ce9e9da0e829ab95bf304e1556f8e13f5011d01bc8a85879b1ef832" exitCode=0 Oct 31 23:15:11.189076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.136731 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:11.189076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.138395 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token\") pod \"e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:11.189076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.165831 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.189076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.166604 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.195224 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.196305 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.197079 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.201712 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.202827 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.203897 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:11.225569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.204927 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:11.438967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.438874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:11.559649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.552866 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:11.670886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.646421 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0193517d-e683-4abb-bd5c-57f53ca059b7" path="/var/lib/kubelet/pods/0193517d-e683-4abb-bd5c-57f53ca059b7/volumes" Oct 31 23:15:11.670886 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:11.647113 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.556s" Oct 31 23:15:11.670886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.647148 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" event={"ID":"c5adf9c0-7d39-422e-8f9b-e64c4572477a","Type":"ContainerDied","Data":"df0c471c5ce9e9da0e829ab95bf304e1556f8e13f5011d01bc8a85879b1ef832"} Oct 31 23:15:11.803875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.686781 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:15:11.841296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:11.841252 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:12.033858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.033811 5043 patch_prober.go:29] interesting pod/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:15:12.034146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.034118 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" podUID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:15:12.110290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.050257 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 31 23:15:12.204603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.111401 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.204603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.111478 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.204603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.140155 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 31 23:15:12.204603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.176226 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:12.303593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.299984 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:12.361621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.336414 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.404571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.387312 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.404571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.401661 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.404571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.401734 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.446733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.420731 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.507599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.507501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:12.531847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.508607 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.531847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.527632 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:12.531847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.529908 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:12.546523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.546388 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:12.553124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.553074 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:12.586519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.567610 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:12.609560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.586948 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.612197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.587007 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:12.613517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.587156 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:15:12.632612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.610264 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.637686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.637631 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.808867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:12.808802 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:12.899621 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:12.875722 5043 logging.go:55] [core] [Channel #161 SubChannel #164]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.144:50052", ServerName: "10.132.0.144:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.144:50052: connect: connection refused" Oct 31 23:15:13.007011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.004652 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.312787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.312728 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 23:15:13.364987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.341625 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.406608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.404928 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:13.406608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.405003 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.481488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.469107 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.619635 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:13.580215 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:15:13.622703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.621123 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.644809 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.645092 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.645283 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.645436 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.645638 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.645966 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.646062 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.646233 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.646386 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.651180 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.651254 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.688219 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" podUID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.144:50052\" within 1s: context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.694045 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.714532 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" podUID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.142:50052\" within 1s: context deadline exceeded" Oct 31 23:15:13.715342 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:13.714744 5043 logging.go:55] [core] [Channel #162 SubChannel #163]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.142:50052", ServerName: "10.132.0.142:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.142:50052: operation was canceled" Oct 31 23:15:13.832572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.807872 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:15:13.832908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.832862 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.844506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.834220 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:13.869905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.858487 5043 generic.go:356] "Generic (PLEG): container finished" podID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" containerID="d5b88374f2d00a8bdda3dc21ec7d90765349a8c790b50f263735665dd191ecd7" exitCode=0 Oct 31 23:15:14.498837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:13.913604 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:14.499517 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:13.944832 5043 logging.go:55] [core] [Channel #168 SubChannel #169]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.143:50052", ServerName: "10.132.0.143:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.143:50052: connect: connection refused" Oct 31 23:15:14.499766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.194854 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg\" (UniqueName: \"kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg\") pod \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " Oct 31 23:15:14.500016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.499999 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume\") pod \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\" (UID: \"c5adf9c0-7d39-422e-8f9b-e64c4572477a\") " Oct 31 23:15:14.520261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.497210 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:14.551306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.551261 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:14.564567 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:14.195743 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.717s" Oct 31 23:15:14.564989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.564957 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" event={"ID":"3d9209e0-2e53-48cb-9855-d711a23ad9e5","Type":"ContainerDied","Data":"d5b88374f2d00a8bdda3dc21ec7d90765349a8c790b50f263735665dd191ecd7"} Oct 31 23:15:14.698145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.698098 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:15:14.716580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.698604 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:14.717181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.717140 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:14.717590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.717566 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:14.717939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.717908 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:14.718259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.718237 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:15:14.718606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.718575 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:14.747382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.747327 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:14.761687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.757855 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:15:14.810566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.793746 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:14.824872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.824831 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume" pod "c5adf9c0-7d39-422e-8f9b-e64c4572477a" (UID: "c5adf9c0-7d39-422e-8f9b-e64c4572477a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:15.257818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.982444 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:15.258057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.621675 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:15:15.258342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.080688 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" podUID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.143:50052\" within 1s: context deadline exceeded" Oct 31 23:15:15.284308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.081753 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:15:15.284702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:14.372412 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:15.284842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.148657 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:15.284949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.148703 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 31 23:15:15.285048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.268527 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5adf9c0-7d39-422e-8f9b-e64c4572477a-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:15.454601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.454561 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg" pod "c5adf9c0-7d39-422e-8f9b-e64c4572477a" (UID: "c5adf9c0-7d39-422e-8f9b-e64c4572477a"). InnerVolumeSpecName "kube-api-access-pslwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:15.858570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.857673 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" Oct 31 23:15:15.898571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:15.870974 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:16.103575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.054009 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:16.127637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.091071 5043 scope.go:122] "RemoveContainer" containerID="9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6" Oct 31 23:15:16.413230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.413194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:16.413730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.413693 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.506276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.506230 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.594445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.594405 5043 generic.go:356] "Generic (PLEG): container finished" podID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" containerID="407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9" exitCode=0 Oct 31 23:15:16.661403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.640680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:16.661743 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:16.651142 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.006s" Oct 31 23:15:16.740569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.724730 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.798571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.741465 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.798571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.741513 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.798571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.741532 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.799000 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.741570 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:16.799421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.741604 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:16.799828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.753205 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pslwg\" (UniqueName: \"kubernetes.io/projected/c5adf9c0-7d39-422e-8f9b-e64c4572477a-kube-api-access-pslwg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:16.799943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.754090 5043 scope.go:122] "RemoveContainer" containerID="4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723" Oct 31 23:15:16.818574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.782032 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:16.852604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.801094 5043 scope.go:122] "RemoveContainer" containerID="812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b" Oct 31 23:15:17.225897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.801147 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:17.226598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.226573 5043 scope.go:122] "RemoveContainer" containerID="ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39" Oct 31 23:15:17.257624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.856926 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:15:17.660968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.871682 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:17.660968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.911273 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:15:17.660968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.190077 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:17.660968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.219985 5043 generic.go:356] "Generic (PLEG): container finished" podID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" containerID="44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50" exitCode=0 Oct 31 23:15:17.660968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:16.856888 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:15:17.782058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.703369 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:17.793649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.793056 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:17.794378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.793856 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:17.809105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.795606 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:17.809105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.803059 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:15:17.809105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.805072 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:17.809105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.805447 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:17.851586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.848154 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9\" (UniqueName: \"kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9\") pod \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " Oct 31 23:15:17.851586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.848259 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume\") pod \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\" (UID: \"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036\") " Oct 31 23:15:17.895019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.888999 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume" pod "66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" (UID: "66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:17.917663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.903342 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:17.960668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:17.960564 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:18.077711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.033640 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:18.077711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.051595 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.355814 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.395725 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.474425 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.542746 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9" pod "66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" (UID: "66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036"). InnerVolumeSpecName "kube-api-access-4hkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.561376 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.603788 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.605402 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:18.611435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.605529 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:18.672956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.651889 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:15:18.672956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.652902 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:15:18.672956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.653288 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:18.672956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.656181 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:18.712944 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:18.676004 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.014s" Oct 31 23:15:18.712944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.710243 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:18.712944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.711531 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.712944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.711602 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.713308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.713184 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:18.759721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.713725 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:18.759721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.722225 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:18.759721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.730410 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkq9\" (UniqueName: \"kubernetes.io/projected/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036-kube-api-access-4hkq9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:18.759721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.730434 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:18.759721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.730449 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.877931 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878023 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume\") pod \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878166 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878264 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878302 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878345 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878436 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr\") pod \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\" (UID: \"97551e11-91d7-421c-aad0-aee5c3c8b8b2\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.878501 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.886108 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:15:18.924924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:18.896901 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf\") pod \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\" (UID: \"3d9209e0-2e53-48cb-9855-d711a23ad9e5\") " Oct 31 23:15:19.086583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.067991 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:15:19.086583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.069603 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:15:19.130570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.113835 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:19.130570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.115246 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:15:19.188400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.186348 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:15:19.211346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.211315 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume" pod "97551e11-91d7-421c-aad0-aee5c3c8b8b2" (UID: "97551e11-91d7-421c-aad0-aee5c3c8b8b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:19.212396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.212369 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:19.221761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.221733 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:19.311414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.311379 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97551e11-91d7-421c-aad0-aee5c3c8b8b2-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.503896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.503870 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.504385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.504355 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.504525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.495069 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:19.504907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.476467 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:15:19.505145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.479134 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:19.505258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.480581 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:19.505337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.488506 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "kube-api-access-rdlxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:19.505412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.498761 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr" pod "97551e11-91d7-421c-aad0-aee5c3c8b8b2" (UID: "97551e11-91d7-421c-aad0-aee5c3c8b8b2"). InnerVolumeSpecName "kube-api-access-44bfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:19.505508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.499492 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy" pod "3d9209e0-2e53-48cb-9855-d711a23ad9e5" (UID: "3d9209e0-2e53-48cb-9855-d711a23ad9e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565697 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565743 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4" event={"ID":"c5adf9c0-7d39-422e-8f9b-e64c4572477a","Type":"ContainerDied","Data":"4955cd987fb043633520f011e6c07b4d700d2a779137219a3684df720c552dbd"} Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565798 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565835 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565866 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" event={"ID":"97551e11-91d7-421c-aad0-aee5c3c8b8b2","Type":"ContainerDied","Data":"407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9"} Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565902 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" event={"ID":"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036","Type":"ContainerDied","Data":"44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50"} Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.565948 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.570514 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:15:19.575374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.570584 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:15:19.576457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.576343 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" event={"ID":"97551e11-91d7-421c-aad0-aee5c3c8b8b2","Type":"ContainerDied","Data":"1d559f1523a92b4815b3675f4663e95fcf60ff1d0eceeca72bbfb74838a91333"} Oct 31 23:15:19.595840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.576775 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7" Oct 31 23:15:19.595840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.577403 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:15:19.595840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.585290 5043 scope.go:122] "RemoveContainer" containerID="df0c471c5ce9e9da0e829ab95bf304e1556f8e13f5011d01bc8a85879b1ef832" Oct 31 23:15:19.609169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.604048 5043 scope.go:122] "RemoveContainer" containerID="ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2" Oct 31 23:15:19.609169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.605146 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:19.609169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.606223 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:15:19.609169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.606271 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:15:19.610965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.610941 5043 scope.go:122] "RemoveContainer" containerID="4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639" Oct 31 23:15:19.654212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.654169 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.662291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.661671 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.662291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.661697 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.662291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.661715 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/97551e11-91d7-421c-aad0-aee5c3c8b8b2-kube-api-access-44bfr\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.662291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.661755 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.662291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.661772 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdlxf\" (UniqueName: \"kubernetes.io/projected/3d9209e0-2e53-48cb-9855-d711a23ad9e5-kube-api-access-rdlxf\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:19.662291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.661594 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:15:19.705266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.703326 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:15:19.737838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.727734 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6"] Oct 31 23:15:19.787598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.768697 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac5ff20f-jjqf6"] Oct 31 23:15:19.787598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.780775 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc"] Oct 31 23:15:19.787598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.781005 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" podUID="ff91cb2a-b941-422f-b17b-549ec736d03f" containerName="istio-proxy" containerID="cri-o://dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244" gracePeriod=30 Oct 31 23:15:19.789710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.789064 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:15:19.789966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.789940 5043 scope.go:122] "RemoveContainer" containerID="407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9" Oct 31 23:15:19.795596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.795572 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb"] Oct 31 23:15:19.892666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.891511 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4"] Oct 31 23:15:19.892666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.891740 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" event={"ID":"66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036","Type":"ContainerDied","Data":"37dbe9b7b186edc30a1fb4405eefae88612bba64c4139400946d7aaecc4f0b6b"} Oct 31 23:15:19.892887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.891786 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" event={"ID":"3d9209e0-2e53-48cb-9855-d711a23ad9e5","Type":"ContainerDied","Data":"e283a2cd870a31cb860814e17a5bf197d13be1af46799e2fc0c45b99db7f60df"} Oct 31 23:15:19.892943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.892800 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" event={"ID":"ac53ef9e-c3b9-41c6-aa58-dcbce759111e","Type":"ContainerStarted","Data":"80d9ec767897f6586b3a95d0fe067380b0cd96aa578c593bf2270f14c73abc23"} Oct 31 23:15:19.892943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.808121 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd" Oct 31 23:15:19.908658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.853366 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:19.908658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.811169 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv" Oct 31 23:15:19.911380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:19.911355 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed6117ae-7475557b99-459b4"] Oct 31 23:15:20.010737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.010713 5043 scope.go:122] "RemoveContainer" containerID="44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50" Oct 31 23:15:20.063618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.058259 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610be94b-a435-4d5c-a1f9-71b5fc66b590" path="/var/lib/kubelet/pods/610be94b-a435-4d5c-a1f9-71b5fc66b590/volumes" Oct 31 23:15:20.082519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.082211 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" path="/var/lib/kubelet/pods/c5adf9c0-7d39-422e-8f9b-e64c4572477a/volumes" Oct 31 23:15:20.083123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.083099 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:20.290798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.288819 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:20.890862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.890777 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" event={"ID":"ac53ef9e-c3b9-41c6-aa58-dcbce759111e","Type":"ContainerStarted","Data":"e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217"} Oct 31 23:15:20.894314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.894297 5043 scope.go:122] "RemoveContainer" containerID="407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9" Oct 31 23:15:20.894838 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.894754 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9\": container with ID starting with 407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9 not found: ID does not exist" containerID="407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9" Oct 31 23:15:20.894838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.894795 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9"} err="failed to get container status \"407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9\": rpc error: code = NotFound desc = could not find container \"407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9\": container with ID starting with 407073f5ef6b3a41f912be6a57f5fbcb28a489290297d2b01100ec225ead7dd9 not found: ID does not exist" Oct 31 23:15:20.894838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.894819 5043 scope.go:122] "RemoveContainer" containerID="44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50" Oct 31 23:15:20.895080 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.895059 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50\": container with ID starting with 44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50 not found: ID does not exist" containerID="44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50" Oct 31 23:15:20.895247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.895224 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50"} err="failed to get container status \"44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50\": rpc error: code = NotFound desc = could not find container \"44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50\": container with ID starting with 44b21ac14f28b170398b5f267edfc174b5ae0cbbebfb6aca0afaca3ee0c21f50 not found: ID does not exist" Oct 31 23:15:20.895337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.895326 5043 scope.go:122] "RemoveContainer" containerID="d5b88374f2d00a8bdda3dc21ec7d90765349a8c790b50f263735665dd191ecd7" Oct 31 23:15:20.898270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.898236 5043 generic.go:356] "Generic (PLEG): container finished" podID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerID="5185457595c38b6dd9fbe475874f0fd00224dc1a723e3abbc2934f3be979e78f" exitCode=60 Oct 31 23:15:20.898486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.898317 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" event={"ID":"b819fdb2-a681-41df-9020-be1f3f0f1e44","Type":"ContainerDied","Data":"5185457595c38b6dd9fbe475874f0fd00224dc1a723e3abbc2934f3be979e78f"} Oct 31 23:15:20.899055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.898966 5043 scope.go:122] "RemoveContainer" containerID="5185457595c38b6dd9fbe475874f0fd00224dc1a723e3abbc2934f3be979e78f" Oct 31 23:15:20.901030 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.901004 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4_odh-ai-gateway-infra(b819fdb2-a681-41df-9020-be1f3f0f1e44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" Oct 31 23:15:20.914052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.908839 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerStarted","Data":"f3e4ec8080cd707ae9399bf74ef991bb805ce09e65387a3cd9b24aa4472c26ed"} Oct 31 23:15:20.914052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.909413 5043 scope.go:122] "RemoveContainer" containerID="f3e4ec8080cd707ae9399bf74ef991bb805ce09e65387a3cd9b24aa4472c26ed" Oct 31 23:15:20.914052 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.909790 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn_odh-ai-gateway-infra(fa0af7e9-5bed-41ab-9cf1-155215ec20f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" Oct 31 23:15:20.964361 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.964309 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003fa55a_443d_4e54_8842_f426d9d46ad5.slice/crio-conmon-e7865a3fa645c57314ecfc69c382af8a96990be79d2b53cd8908bb303692a54f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96d5f04_c004_4d54_836a_f040ff8b99ca.slice/crio-conmon-0ae294094182167612593ffd1ac12cc1de68f8bb09bd61e1732f6f88a32a2dce.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96d5f04_c004_4d54_836a_f040ff8b99ca.slice/crio-0ae294094182167612593ffd1ac12cc1de68f8bb09bd61e1732f6f88a32a2dce.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003fa55a_443d_4e54_8842_f426d9d46ad5.slice/crio-e7865a3fa645c57314ecfc69c382af8a96990be79d2b53cd8908bb303692a54f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 23:15:20.968740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.968646 5043 scope.go:122] "RemoveContainer" containerID="9b5a00c201b85fc4279428af14b3e3d7ae8323c951a4fb8d0b83f003611527d6" Oct 31 23:15:20.976359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.976331 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerStarted","Data":"c706220a9ff260f2b39c8b9d059cf3f93ed6c96aebe12bc7f326475f5ee590f1"} Oct 31 23:15:20.984356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.984329 5043 generic.go:356] "Generic (PLEG): container finished" podID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerID="0ae294094182167612593ffd1ac12cc1de68f8bb09bd61e1732f6f88a32a2dce" exitCode=60 Oct 31 23:15:20.984586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.984528 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" event={"ID":"e96d5f04-c004-4d54-836a-f040ff8b99ca","Type":"ContainerDied","Data":"0ae294094182167612593ffd1ac12cc1de68f8bb09bd61e1732f6f88a32a2dce"} Oct 31 23:15:20.984967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.984944 5043 scope.go:122] "RemoveContainer" containerID="0ae294094182167612593ffd1ac12cc1de68f8bb09bd61e1732f6f88a32a2dce" Oct 31 23:15:20.985305 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.985256 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k_odh-ai-gateway-infra(e96d5f04-c004-4d54-836a-f040ff8b99ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" Oct 31 23:15:20.988074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.988050 5043 generic.go:356] "Generic (PLEG): container finished" podID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerID="e7865a3fa645c57314ecfc69c382af8a96990be79d2b53cd8908bb303692a54f" exitCode=60 Oct 31 23:15:20.988161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.988127 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" event={"ID":"003fa55a-443d-4e54-8842-f426d9d46ad5","Type":"ContainerDied","Data":"e7865a3fa645c57314ecfc69c382af8a96990be79d2b53cd8908bb303692a54f"} Oct 31 23:15:20.988516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.988497 5043 scope.go:122] "RemoveContainer" containerID="e7865a3fa645c57314ecfc69c382af8a96990be79d2b53cd8908bb303692a54f" Oct 31 23:15:20.988847 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:20.988823 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891475-h5sln_odh-ai-gateway-infra(003fa55a-443d-4e54-8842-f426d9d46ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" Oct 31 23:15:20.990461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:20.990444 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerStarted","Data":"781aa091caf21cf22ec7bdf716fc324d4d9959091519e464ac6339f8b858a8dd"} Oct 31 23:15:21.005794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.005774 5043 scope.go:122] "RemoveContainer" containerID="812a92873cf9c21ed167c1bcbd0adfedd67ffde56316e0f758220a311c908a8b" Oct 31 23:15:21.008585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.006006 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv"] Oct 31 23:15:21.070582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.069815 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b2c6553b-openshift-default-79d994787c-bbstv"] Oct 31 23:15:21.071863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.071832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:21.089234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.088867 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7"] Oct 31 23:15:21.095671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.091027 5043 scope.go:122] "RemoveContainer" containerID="ff595aae006b8478f09b5bd496a870b348c0b8aa5667a09f7fa3eb0bb951cb39" Oct 31 23:15:21.099484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.099462 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2c6553b-5f49f58758-jrvn7"] Oct 31 23:15:21.119622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.119589 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd"] Oct 31 23:15:21.282981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.281109 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:21.517628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.497939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:21.591572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.585766 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:21.677926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.677894 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2c6553b-647f7fb6d7-nmgcd"] Oct 31 23:15:21.882142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.882057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:21.885673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.885060 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" podStartSLOduration=13.885043188000001 podStartE2EDuration="13.885043188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:20.969153214 +0000 UTC m=+1547.198707629" watchObservedRunningTime="2026-10-31 23:15:21.885043188 +0000 UTC m=+1548.114597598" Oct 31 23:15:21.913860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.913802 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" podStartSLOduration=21.913786246 podStartE2EDuration="21.913786246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:21.070018321 +0000 UTC m=+1547.299572732" watchObservedRunningTime="2026-10-31 23:15:21.913786246 +0000 UTC m=+1548.143340658" Oct 31 23:15:21.961839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.961795 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" path="/var/lib/kubelet/pods/3d9209e0-2e53-48cb-9855-d711a23ad9e5/volumes" Oct 31 23:15:21.962628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.962602 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" path="/var/lib/kubelet/pods/66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036/volumes" Oct 31 23:15:21.963219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:21.963197 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" path="/var/lib/kubelet/pods/97551e11-91d7-421c-aad0-aee5c3c8b8b2/volumes" Oct 31 23:15:22.005670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.005635 5043 generic.go:356] "Generic (PLEG): container finished" podID="99c12441-fdb1-4920-aeef-3a351971a5df" containerID="781aa091caf21cf22ec7bdf716fc324d4d9959091519e464ac6339f8b858a8dd" exitCode=60 Oct 31 23:15:22.005816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.005778 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerDied","Data":"781aa091caf21cf22ec7bdf716fc324d4d9959091519e464ac6339f8b858a8dd"} Oct 31 23:15:22.005852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.005823 5043 scope.go:122] "RemoveContainer" containerID="4215bac465d9f1fb21e836906d437e5ac98b27f3600b05dbd56ecc61ef4bd639" Oct 31 23:15:22.006149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.006135 5043 scope.go:122] "RemoveContainer" containerID="781aa091caf21cf22ec7bdf716fc324d4d9959091519e464ac6339f8b858a8dd" Oct 31 23:15:22.006431 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:22.006412 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr_odh-ai-gateway-infra(99c12441-fdb1-4920-aeef-3a351971a5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" Oct 31 23:15:22.012328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.012299 5043 generic.go:356] "Generic (PLEG): container finished" podID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerID="f3e4ec8080cd707ae9399bf74ef991bb805ce09e65387a3cd9b24aa4472c26ed" exitCode=60 Oct 31 23:15:22.012444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.012415 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerDied","Data":"f3e4ec8080cd707ae9399bf74ef991bb805ce09e65387a3cd9b24aa4472c26ed"} Oct 31 23:15:22.013276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.013250 5043 scope.go:122] "RemoveContainer" containerID="f3e4ec8080cd707ae9399bf74ef991bb805ce09e65387a3cd9b24aa4472c26ed" Oct 31 23:15:22.013641 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:22.013615 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn_odh-ai-gateway-infra(fa0af7e9-5bed-41ab-9cf1-155215ec20f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" Oct 31 23:15:22.015143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.015118 5043 generic.go:356] "Generic (PLEG): container finished" podID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerID="c706220a9ff260f2b39c8b9d059cf3f93ed6c96aebe12bc7f326475f5ee590f1" exitCode=60 Oct 31 23:15:22.015308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.015291 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerDied","Data":"c706220a9ff260f2b39c8b9d059cf3f93ed6c96aebe12bc7f326475f5ee590f1"} Oct 31 23:15:22.017800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.017775 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4"] Oct 31 23:15:22.019327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.019309 5043 scope.go:122] "RemoveContainer" containerID="c706220a9ff260f2b39c8b9d059cf3f93ed6c96aebe12bc7f326475f5ee590f1" Oct 31 23:15:22.019774 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:22.019751 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj_odh-ai-gateway-infra(3f788bf6-0c66-4d64-9495-b46fb49037d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" Oct 31 23:15:22.032737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.032718 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:22.034781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.034755 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:15:22.039742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.039665 5043 scope.go:122] "RemoveContainer" containerID="4a2bd9ddb53ce9acbf4a0bfb39026c0279ac9bcd99f6d054420db5c504fcc723" Oct 31 23:15:22.047617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.047597 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn"] Oct 31 23:15:22.079957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.079939 5043 scope.go:122] "RemoveContainer" containerID="ed2ab20a9cd8e590ff189968fc2cb5c98807f7837242c00e4cf008352f9b4bc2" Oct 31 23:15:22.106696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.106665 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr"] Oct 31 23:15:22.209567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.173873 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k"] Oct 31 23:15:22.209567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.198770 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj"] Oct 31 23:15:22.346629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.344882 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln"] Oct 31 23:15:22.489428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.489001 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:22.570527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.567905 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:22.613242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.613177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:22.921593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.914100 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:22.943426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.941188 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca\") pod \"b819fdb2-a681-41df-9020-be1f3f0f1e44\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " Oct 31 23:15:22.943426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.941413 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca\") pod \"e96d5f04-c004-4d54-836a-f040ff8b99ca\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " Oct 31 23:15:22.943426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.941480 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6\" (UniqueName: \"kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6\") pod \"b819fdb2-a681-41df-9020-be1f3f0f1e44\" (UID: \"b819fdb2-a681-41df-9020-be1f3f0f1e44\") " Oct 31 23:15:22.943426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.941639 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8\" (UniqueName: \"kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8\") pod \"e96d5f04-c004-4d54-836a-f040ff8b99ca\" (UID: \"e96d5f04-c004-4d54-836a-f040ff8b99ca\") " Oct 31 23:15:22.950100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.949311 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8" pod "e96d5f04-c004-4d54-836a-f040ff8b99ca" (UID: "e96d5f04-c004-4d54-836a-f040ff8b99ca"). InnerVolumeSpecName "kube-api-access-9glw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:22.950100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.949715 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca" pod "b819fdb2-a681-41df-9020-be1f3f0f1e44" (UID: "b819fdb2-a681-41df-9020-be1f3f0f1e44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:22.964759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.964029 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca" pod "e96d5f04-c004-4d54-836a-f040ff8b99ca" (UID: "e96d5f04-c004-4d54-836a-f040ff8b99ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:22.987218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.987180 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6" pod "b819fdb2-a681-41df-9020-be1f3f0f1e44" (UID: "b819fdb2-a681-41df-9020-be1f3f0f1e44"). InnerVolumeSpecName "kube-api-access-sdcf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:22.988893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:22.988858 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:23.048107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.048072 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca\") pod \"003fa55a-443d-4e54-8842-f426d9d46ad5\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " Oct 31 23:15:23.048362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.048342 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh\" (UniqueName: \"kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh\") pod \"003fa55a-443d-4e54-8842-f426d9d46ad5\" (UID: \"003fa55a-443d-4e54-8842-f426d9d46ad5\") " Oct 31 23:15:23.049380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.049353 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d5f04-c004-4d54-836a-f040ff8b99ca-maas-api-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.050223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.050206 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcf6\" (UniqueName: \"kubernetes.io/projected/b819fdb2-a681-41df-9020-be1f3f0f1e44-kube-api-access-sdcf6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.050373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.050361 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9glw8\" (UniqueName: \"kubernetes.io/projected/e96d5f04-c004-4d54-836a-f040ff8b99ca-kube-api-access-9glw8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.050453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.050442 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b819fdb2-a681-41df-9020-be1f3f0f1e44-maas-api-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.050524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.050166 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca" pod "003fa55a-443d-4e54-8842-f426d9d46ad5" (UID: "003fa55a-443d-4e54-8842-f426d9d46ad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:23.053225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.053195 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" event={"ID":"b819fdb2-a681-41df-9020-be1f3f0f1e44","Type":"ContainerDied","Data":"3f5dc6959e4009241a04df7e5c3e3ede74fc2cb1d96cbf848ff9bd209a3851df"} Oct 31 23:15:23.053390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.053377 5043 scope.go:122] "RemoveContainer" containerID="5185457595c38b6dd9fbe475874f0fd00224dc1a723e3abbc2934f3be979e78f" Oct 31 23:15:23.053632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.053615 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4" Oct 31 23:15:23.058304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.058283 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh" pod "003fa55a-443d-4e54-8842-f426d9d46ad5" (UID: "003fa55a-443d-4e54-8842-f426d9d46ad5"). InnerVolumeSpecName "kube-api-access-wj6mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:23.172350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.172252 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj6mh\" (UniqueName: \"kubernetes.io/projected/003fa55a-443d-4e54-8842-f426d9d46ad5-kube-api-access-wj6mh\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.172350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.172291 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003fa55a-443d-4e54-8842-f426d9d46ad5-maas-api-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.217613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.217515 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" event={"ID":"e96d5f04-c004-4d54-836a-f040ff8b99ca","Type":"ContainerDied","Data":"fa6548dc76db8f2b0dd258f2283b14dfdece26ccd32cdb57429833173f3de06f"} Oct 31 23:15:23.217756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.217712 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k" Oct 31 23:15:23.231899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.230229 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" event={"ID":"003fa55a-443d-4e54-8842-f426d9d46ad5","Type":"ContainerDied","Data":"1717922d27630502b9535d2586dfd7944e0e54a140d0ec5217f615f9d10a0f81"} Oct 31 23:15:23.231899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.230340 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln" Oct 31 23:15:23.250475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.248616 5043 scope.go:122] "RemoveContainer" containerID="0ae294094182167612593ffd1ac12cc1de68f8bb09bd61e1732f6f88a32a2dce" Oct 31 23:15:23.258330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.257760 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4"] Oct 31 23:15:23.272369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.272314 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-255748-29891475-vjwm4"] Oct 31 23:15:23.292159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.292125 5043 scope.go:122] "RemoveContainer" containerID="e7865a3fa645c57314ecfc69c382af8a96990be79d2b53cd8908bb303692a54f" Oct 31 23:15:23.354956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.354910 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln"] Oct 31 23:15:23.366600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.364498 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891475-h5sln"] Oct 31 23:15:23.381874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.381813 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k"] Oct 31 23:15:23.393528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.393444 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2294a7-29891475-zlg2k"] Oct 31 23:15:23.500928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.500074 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:23.586931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.586112 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj\" (UniqueName: \"kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj\") pod \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " Oct 31 23:15:23.586931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.586288 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca\") pod \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\" (UID: \"fa0af7e9-5bed-41ab-9cf1-155215ec20f1\") " Oct 31 23:15:23.587299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.587275 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca" pod "fa0af7e9-5bed-41ab-9cf1-155215ec20f1" (UID: "fa0af7e9-5bed-41ab-9cf1-155215ec20f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:23.590735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.590707 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj" pod "fa0af7e9-5bed-41ab-9cf1-155215ec20f1" (UID: "fa0af7e9-5bed-41ab-9cf1-155215ec20f1"). InnerVolumeSpecName "kube-api-access-bx6zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:23.590967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.590948 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:23.658248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.657807 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:23.662333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.661401 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:23.741991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.731741 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx\" (UniqueName: \"kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx\") pod \"3f788bf6-0c66-4d64-9495-b46fb49037d7\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " Oct 31 23:15:23.741991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.731894 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca\") pod \"3f788bf6-0c66-4d64-9495-b46fb49037d7\" (UID: \"3f788bf6-0c66-4d64-9495-b46fb49037d7\") " Oct 31 23:15:23.741991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.732768 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6zj\" (UniqueName: \"kubernetes.io/projected/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-kube-api-access-bx6zj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.741991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.732787 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0af7e9-5bed-41ab-9cf1-155215ec20f1-maas-api-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.741991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.739069 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx" pod "3f788bf6-0c66-4d64-9495-b46fb49037d7" (UID: "3f788bf6-0c66-4d64-9495-b46fb49037d7"). InnerVolumeSpecName "kube-api-access-jz4sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:23.742594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.742533 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca" pod "3f788bf6-0c66-4d64-9495-b46fb49037d7" (UID: "3f788bf6-0c66-4d64-9495-b46fb49037d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:23.835186 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.834250 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca\") pod \"99c12441-fdb1-4920-aeef-3a351971a5df\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " Oct 31 23:15:23.835186 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.834307 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg\" (UniqueName: \"kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg\") pod \"99c12441-fdb1-4920-aeef-3a351971a5df\" (UID: \"99c12441-fdb1-4920-aeef-3a351971a5df\") " Oct 31 23:15:23.835186 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.835147 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca" pod "99c12441-fdb1-4920-aeef-3a351971a5df" (UID: "99c12441-fdb1-4920-aeef-3a351971a5df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:23.835530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.835488 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4sx\" (UniqueName: \"kubernetes.io/projected/3f788bf6-0c66-4d64-9495-b46fb49037d7-kube-api-access-jz4sx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.835530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.835517 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c12441-fdb1-4920-aeef-3a351971a5df-maas-api-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.835670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.835532 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f788bf6-0c66-4d64-9495-b46fb49037d7-maas-api-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.837313 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.837285 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg" pod "99c12441-fdb1-4920-aeef-3a351971a5df" (UID: "99c12441-fdb1-4920-aeef-3a351971a5df"). InnerVolumeSpecName "kube-api-access-9rppg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:23.863058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.863025 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:23.877086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877057 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j"] Oct 31 23:15:23.877802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877786 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877803 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877812 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877817 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877829 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" containerName="payload-pre-processing" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877835 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" containerName="payload-pre-processing" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877849 5043 cpu_manager.go:414] "containerMap: removing container" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:23.877859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877854 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877864 5043 cpu_manager.go:414] "containerMap: removing container" podUID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" containerName="payload-pre-processing" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877869 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" containerName="payload-pre-processing" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877877 5043 cpu_manager.go:414] "containerMap: removing container" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877882 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877889 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877893 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877905 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877909 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877917 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877921 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877928 5043 cpu_manager.go:414] "containerMap: removing container" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877932 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877939 5043 cpu_manager.go:414] "containerMap: removing container" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877944 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877955 5043 cpu_manager.go:414] "containerMap: removing container" podUID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" containerName="payload-processing" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877959 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" containerName="payload-processing" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877965 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877969 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877978 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877982 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877994 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.877998 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878004 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878008 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878017 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878022 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878029 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" containerName="istio-proxy" Oct 31 23:15:23.878073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878033 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" containerName="istio-proxy" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878138 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878145 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="66aa7ac3-0ebb-4b0d-ab84-33d8b51b3036" containerName="payload-pre-processing" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878153 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878159 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878165 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878170 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878177 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878181 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878188 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="97551e11-91d7-421c-aad0-aee5c3c8b8b2" containerName="payload-processing" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878192 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878197 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878204 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878208 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878213 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878219 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9209e0-2e53-48cb-9855-d711a23ad9e5" containerName="istio-proxy" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878225 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878232 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:23.878924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.878236 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5adf9c0-7d39-422e-8f9b-e64c4572477a" containerName="payload-pre-processing" Oct 31 23:15:23.885214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.885185 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:23.892051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.892018 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j"] Oct 31 23:15:23.898960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.898452 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-907d182f\"" Oct 31 23:15:23.903854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.903839 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-907d182f-dockercfg-zs9km\"" Oct 31 23:15:23.950108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.947867 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rppg\" (UniqueName: \"kubernetes.io/projected/99c12441-fdb1-4920-aeef-3a351971a5df-kube-api-access-9rppg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:23.980683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.980651 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" path="/var/lib/kubelet/pods/003fa55a-443d-4e54-8842-f426d9d46ad5/volumes" Oct 31 23:15:23.981655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.981625 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" path="/var/lib/kubelet/pods/b819fdb2-a681-41df-9020-be1f3f0f1e44/volumes" Oct 31 23:15:23.983080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:23.983054 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" path="/var/lib/kubelet/pods/e96d5f04-c004-4d54-836a-f040ff8b99ca/volumes" Oct 31 23:15:24.054812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.022377 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:24.054812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.051271 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgdp\" (UniqueName: \"kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp\") pod \"payload-processing-e2e-ait-907d182f-648bf98f76-hb76j\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.054812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.051642 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume\") pod \"payload-processing-e2e-ait-907d182f-648bf98f76-hb76j\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.177567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.169167 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgdp\" (UniqueName: \"kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp\") pod \"payload-processing-e2e-ait-907d182f-648bf98f76-hb76j\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.177567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.169598 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume\") pod \"payload-processing-e2e-ait-907d182f-648bf98f76-hb76j\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.177567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.171068 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume\") pod \"payload-processing-e2e-ait-907d182f-648bf98f76-hb76j\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.229615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.228121 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgdp\" (UniqueName: \"kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp\") pod \"payload-processing-e2e-ait-907d182f-648bf98f76-hb76j\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.368580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.368522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" event={"ID":"99c12441-fdb1-4920-aeef-3a351971a5df","Type":"ContainerDied","Data":"afcfe26073976a51a98181dacd8ce8472ff2541be0dd2be8c398d073ce7b2d6b"} Oct 31 23:15:24.368773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.368615 5043 scope.go:122] "RemoveContainer" containerID="781aa091caf21cf22ec7bdf716fc324d4d9959091519e464ac6339f8b858a8dd" Oct 31 23:15:24.373818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.373797 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" Oct 31 23:15:24.375329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.375302 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn" event={"ID":"fa0af7e9-5bed-41ab-9cf1-155215ec20f1","Type":"ContainerDied","Data":"231d195df06e3ff056c63a2be126657cb7f31abf829638f71b1914a18194185d"} Oct 31 23:15:24.378167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.378095 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" event={"ID":"3f788bf6-0c66-4d64-9495-b46fb49037d7","Type":"ContainerDied","Data":"377db94d1316d3f4ec88e9c641edf1e25aa2e053df660cb6e8326c750779ba00"} Oct 31 23:15:24.389492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.378454 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj" Oct 31 23:15:24.389492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.368767 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr" Oct 31 23:15:24.389775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.389535 5043 scope.go:122] "RemoveContainer" containerID="f3e4ec8080cd707ae9399bf74ef991bb805ce09e65387a3cd9b24aa4472c26ed" Oct 31 23:15:24.405511 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.405191 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn"] Oct 31 23:15:24.409308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.409271 5043 scope.go:122] "RemoveContainer" containerID="c706220a9ff260f2b39c8b9d059cf3f93ed6c96aebe12bc7f326475f5ee590f1" Oct 31 23:15:24.413722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.413696 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a16dfb-29891475-phbsn"] Oct 31 23:15:24.434429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.434371 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj"] Oct 31 23:15:24.447469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.447440 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-04c5c2-29891475-c5fcj"] Oct 31 23:15:24.458631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.458605 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr"] Oct 31 23:15:24.470112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.470082 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-67fce9-29891475-lv8xr"] Oct 31 23:15:24.506125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.506095 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:24.708971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.708926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:24.848441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.848393 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j"] Oct 31 23:15:24.871592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:24.870107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:25.399258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:25.399214 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" event={"ID":"be07e464-5d0a-4050-8ae7-49f904f56811","Type":"ContainerStarted","Data":"1630c2c3fdbc7510ad1db16422ed42a10aee8600eb28758b124ab096424dffd7"} Oct 31 23:15:25.399968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:25.399271 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" event={"ID":"be07e464-5d0a-4050-8ae7-49f904f56811","Type":"ContainerStarted","Data":"e95e86abfdb76e3b4c296cffbfc505f1551a875b061cb8faca0c13eb2c619ae3"} Oct 31 23:15:25.399968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:25.399308 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:25.424808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:25.424755 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" podStartSLOduration=2.424741241 podStartE2EDuration="2.424741241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:25.417860576 +0000 UTC m=+1551.647414986" watchObservedRunningTime="2026-10-31 23:15:25.424741241 +0000 UTC m=+1551.654295651" Oct 31 23:15:25.635190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:25.634794 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:26.076396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:26.076353 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" path="/var/lib/kubelet/pods/3f788bf6-0c66-4d64-9495-b46fb49037d7/volumes" Oct 31 23:15:26.089568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:26.079425 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" path="/var/lib/kubelet/pods/99c12441-fdb1-4920-aeef-3a351971a5df/volumes" Oct 31 23:15:26.124583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:26.113238 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0af7e9-5bed-41ab-9cf1-155215ec20f1" path="/var/lib/kubelet/pods/fa0af7e9-5bed-41ab-9cf1-155215ec20f1/volumes" Oct 31 23:15:26.412959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:26.412866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:27.500086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:27.500044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:27.663081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:27.663048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:28.309066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:28.309026 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:29.101141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.101068 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:29.436366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.436340 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:15:29.484180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.484146 5043 generic.go:356] "Generic (PLEG): container finished" podID="ff91cb2a-b941-422f-b17b-549ec736d03f" containerID="dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244" exitCode=0 Oct 31 23:15:29.484371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.484200 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" Oct 31 23:15:29.484371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.484222 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" event={"ID":"ff91cb2a-b941-422f-b17b-549ec736d03f","Type":"ContainerDied","Data":"dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244"} Oct 31 23:15:29.484371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.484296 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc" event={"ID":"ff91cb2a-b941-422f-b17b-549ec736d03f","Type":"ContainerDied","Data":"4c9c5441394beb6e9050e7dd89f746fb4674569441d53d2a734148cbacba3c81"} Oct 31 23:15:29.484371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.484327 5043 scope.go:122] "RemoveContainer" containerID="dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244" Oct 31 23:15:29.497427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.497407 5043 scope.go:122] "RemoveContainer" containerID="dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244" Oct 31 23:15:29.497748 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:29.497721 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244\": container with ID starting with dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244 not found: ID does not exist" containerID="dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244" Oct 31 23:15:29.497825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.497761 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244"} err="failed to get container status \"dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244\": rpc error: code = NotFound desc = could not find container \"dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244\": container with ID starting with dc8015a1394ded337cb326d18d7830517163b51cef6d09f5ef4b0f20c4fa5244 not found: ID does not exist" Oct 31 23:15:29.563045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563015 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563081 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563107 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563166 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563188 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563296 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563359 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:29.563458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563450 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563472 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:29.563611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563496 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563607 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token\") pod \"ff91cb2a-b941-422f-b17b-549ec736d03f\" (UID: \"ff91cb2a-b941-422f-b17b-549ec736d03f\") " Oct 31 23:15:29.563778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563651 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:29.563778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563719 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:29.563778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.563739 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:29.564642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.564620 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.564753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.564645 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff91cb2a-b941-422f-b17b-549ec736d03f-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.564753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.564671 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.564753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.564684 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.564753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.564699 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.565577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.565525 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:15:29.565683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.565640 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "kube-api-access-fsbn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:29.565767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.565749 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:15:29.565883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.565868 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token" pod "ff91cb2a-b941-422f-b17b-549ec736d03f" (UID: "ff91cb2a-b941-422f-b17b-549ec736d03f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:29.666136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.666059 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.666136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.666087 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.666136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.666097 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbn6\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-kube-api-access-fsbn6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.666136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.666106 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff91cb2a-b941-422f-b17b-549ec736d03f-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:29.779780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.779707 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:29.823713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.823663 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc"] Oct 31 23:15:29.849330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.849289 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ac5ff20f-openshift-default-758c897847-slrfc"] Oct 31 23:15:29.940754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:29.940675 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff91cb2a-b941-422f-b17b-549ec736d03f" path="/var/lib/kubelet/pods/ff91cb2a-b941-422f-b17b-549ec736d03f/volumes" Oct 31 23:15:30.386306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:30.386270 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:30.539242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:30.539187 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:30.673108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:30.673018 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:31.045336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:31.045305 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx"] Oct 31 23:15:31.056273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:31.056244 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532b7399-zqpmx"] Oct 31 23:15:31.288001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:31.287970 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:31.930896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:31.930846 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:31.961368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:31.961333 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace8520f-c240-4ab4-ae90-20a1be000980" path="/var/lib/kubelet/pods/ace8520f-c240-4ab4-ae90-20a1be000980/volumes" Oct 31 23:15:32.613524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:32.613490 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:33.390217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:33.390181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:33.800780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:33.800737 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:34.221413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:34.221217 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:34.928876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:34.928813 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:35.814796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:35.814764 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:36.415692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:36.415659 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:15:36.513850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:36.513818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:37.200575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:37.198338 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:37.822569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:37.818377 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:38.433970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:38.433938 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:39.045295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.045257 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:39.661837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.661796 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml"] Oct 31 23:15:39.663091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663063 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ff91cb2a-b941-422f-b17b-549ec736d03f" containerName="istio-proxy" Oct 31 23:15:39.663091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663090 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff91cb2a-b941-422f-b17b-549ec736d03f" containerName="istio-proxy" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663137 5043 cpu_manager.go:414] "containerMap: removing container" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663144 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663155 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663163 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663183 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663189 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663208 5043 cpu_manager.go:414] "containerMap: removing container" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:39.663290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663215 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c12441-fdb1-4920-aeef-3a351971a5df" containerName="cleanup" Oct 31 23:15:39.663671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663394 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="003fa55a-443d-4e54-8842-f426d9d46ad5" containerName="cleanup" Oct 31 23:15:39.663671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663409 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96d5f04-c004-4d54-836a-f040ff8b99ca" containerName="cleanup" Oct 31 23:15:39.663671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663422 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b819fdb2-a681-41df-9020-be1f3f0f1e44" containerName="cleanup" Oct 31 23:15:39.663671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663438 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f788bf6-0c66-4d64-9495-b46fb49037d7" containerName="cleanup" Oct 31 23:15:39.663671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.663450 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff91cb2a-b941-422f-b17b-549ec736d03f" containerName="istio-proxy" Oct 31 23:15:39.670083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.670059 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.670805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.670774 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml"] Oct 31 23:15:39.674360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.674288 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:15:39.674483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.674370 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:15:39.675833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.675813 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:15:39.675951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.675872 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:15:39.728129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.728094 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:39.794649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.794585 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.794930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.794711 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.794930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.794827 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8lbh\" (UniqueName: \"kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.896873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.896831 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.897065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.896935 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.897065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.897050 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8lbh\" (UniqueName: \"kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.897687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.897666 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.899217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.899195 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:39.909572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:39.909535 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8lbh\" (UniqueName: \"kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh\") pod \"maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:40.007728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:40.007681 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:40.296724 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:40.296691 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba53702_6261_456b_9e35_790b8ec54e44.slice/crio-387f16ecb61a6ec36be71b444d86a422066e2ece11317e1d58fe1c71e8f7dab6 WatchSource:0}: Error finding container 387f16ecb61a6ec36be71b444d86a422066e2ece11317e1d58fe1c71e8f7dab6: Status 404 returned error can't find the container with id 387f16ecb61a6ec36be71b444d86a422066e2ece11317e1d58fe1c71e8f7dab6 Oct 31 23:15:40.298655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:40.298621 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml"] Oct 31 23:15:40.369655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:40.369626 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:40.460616 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:40.460574 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:15:40.590494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:40.590210 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:40.661790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:40.661747 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" event={"ID":"4ba53702-6261-456b-9e35-790b8ec54e44","Type":"ContainerStarted","Data":"387f16ecb61a6ec36be71b444d86a422066e2ece11317e1d58fe1c71e8f7dab6"} Oct 31 23:15:41.228085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:41.228050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:41.672798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:41.672761 5043 generic.go:356] "Generic (PLEG): container finished" podID="4ba53702-6261-456b-9e35-790b8ec54e44" containerID="fd08310dfb31c18534e922f8f739bac67016207d1fe8942b252a3e7fd51cbd70" exitCode=0 Oct 31 23:15:41.673217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:41.672847 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" event={"ID":"4ba53702-6261-456b-9e35-790b8ec54e44","Type":"ContainerDied","Data":"fd08310dfb31c18534e922f8f739bac67016207d1fe8942b252a3e7fd51cbd70"} Oct 31 23:15:41.938623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:41.937216 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:42.552247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.552184 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:42.885332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.885308 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:42.974383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.974352 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca\") pod \"4ba53702-6261-456b-9e35-790b8ec54e44\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " Oct 31 23:15:42.974563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.974428 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca\") pod \"4ba53702-6261-456b-9e35-790b8ec54e44\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " Oct 31 23:15:42.974563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.974478 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh\" (UniqueName: \"kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh\") pod \"4ba53702-6261-456b-9e35-790b8ec54e44\" (UID: \"4ba53702-6261-456b-9e35-790b8ec54e44\") " Oct 31 23:15:42.974786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.974749 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca" pod "4ba53702-6261-456b-9e35-790b8ec54e44" (UID: "4ba53702-6261-456b-9e35-790b8ec54e44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:42.974989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.974972 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:42.976473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.976451 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca" pod "4ba53702-6261-456b-9e35-790b8ec54e44" (UID: "4ba53702-6261-456b-9e35-790b8ec54e44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:42.977092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:42.977067 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh" pod "4ba53702-6261-456b-9e35-790b8ec54e44" (UID: "4ba53702-6261-456b-9e35-790b8ec54e44"). InnerVolumeSpecName "kube-api-access-x8lbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:43.077220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.077188 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8lbh\" (UniqueName: \"kubernetes.io/projected/4ba53702-6261-456b-9e35-790b8ec54e44-kube-api-access-x8lbh\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:43.077220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.077220 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba53702-6261-456b-9e35-790b8ec54e44-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:43.223616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.222804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:43.694517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.694486 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" Oct 31 23:15:43.694690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.694485 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml" event={"ID":"4ba53702-6261-456b-9e35-790b8ec54e44","Type":"ContainerDied","Data":"387f16ecb61a6ec36be71b444d86a422066e2ece11317e1d58fe1c71e8f7dab6"} Oct 31 23:15:43.694690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.694652 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387f16ecb61a6ec36be71b444d86a422066e2ece11317e1d58fe1c71e8f7dab6" Oct 31 23:15:43.794527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:43.794477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:44.029535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:44.029502 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm"] Oct 31 23:15:44.035958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:44.035933 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720e97ed-fvxcm"] Oct 31 23:15:44.398524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:44.398395 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:45.056624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.056588 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r"] Oct 31 23:15:45.085145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.085111 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5890f43-z598r"] Oct 31 23:15:45.107519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.106276 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87"] Oct 31 23:15:45.143583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.143508 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt"] Oct 31 23:15:45.207910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.207877 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-881f57fe-g4xwt"] Oct 31 23:15:45.245851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.245577 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56b98fe-lmv87"] Oct 31 23:15:45.254438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.254409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:45.956283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.955245 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227acc28-b8eb-4ab9-a764-83b71dc868e8" path="/var/lib/kubelet/pods/227acc28-b8eb-4ab9-a764-83b71dc868e8/volumes" Oct 31 23:15:45.956283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.956019 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e60aa67-7463-4fe2-8154-2ca468cbd804" path="/var/lib/kubelet/pods/7e60aa67-7463-4fe2-8154-2ca468cbd804/volumes" Oct 31 23:15:45.957857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.956710 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9578f643-4094-4d1d-be0d-5493f292a27d" path="/var/lib/kubelet/pods/9578f643-4094-4d1d-be0d-5493f292a27d/volumes" Oct 31 23:15:45.957857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:45.957593 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe928e72-e3a3-484c-8b4e-aa642aed31fc" path="/var/lib/kubelet/pods/fe928e72-e3a3-484c-8b4e-aa642aed31fc/volumes" Oct 31 23:15:46.240626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:46.232693 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:47.380565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:47.379416 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:48.042727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:48.042634 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt"] Oct 31 23:15:48.050737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:48.050701 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be0f84a-spzgt"] Oct 31 23:15:48.122139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:48.122057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:49.050639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.050601 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv"] Oct 31 23:15:49.099570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.055775 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4ba53702-6261-456b-9e35-790b8ec54e44" containerName="revoke-keys" Oct 31 23:15:49.516656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.508065 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba53702-6261-456b-9e35-790b8ec54e44" containerName="revoke-keys" Oct 31 23:15:49.516656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.508397 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba53702-6261-456b-9e35-790b8ec54e44" containerName="revoke-keys" Oct 31 23:15:49.519757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.231865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:49.545198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.545169 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.579896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.578814 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:15:49.579896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.579430 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:15:49.579896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.579533 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:15:49.579896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.579622 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:15:49.620665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.620634 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.621020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.620998 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.621211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.621192 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwt4\" (UniqueName: \"kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.644235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.644206 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv"] Oct 31 23:15:49.730171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.729120 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.730171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.729238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwt4\" (UniqueName: \"kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.730171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.729499 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.752530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.752502 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.775060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.774992 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.777921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.777869 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwt4\" (UniqueName: \"kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4\") pod \"maas-api-revoke-keys-e2e-ait-907d182f-59snv\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:49.974575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:49.958193 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:50.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.037866 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:50.218612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.206759 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:50.286564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.281376 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:50.286564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.282804 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:50.303321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.302480 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:15:50.308149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.308094 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:50.343042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.342950 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:15:50.410220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.410167 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:15:50.424791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.423213 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:50.508209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.508167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:50.537579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.537377 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:15:50.602618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.593180 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eeddfa8-e552-4ed6-a6d7-faeca208530a" path="/var/lib/kubelet/pods/5eeddfa8-e552-4ed6-a6d7-faeca208530a/volumes" Oct 31 23:15:50.702569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.697391 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:50.742188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:50.732576 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:51.335993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.308653 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:15:51.497222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.441173 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:15:51.497222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.492311 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:51.536576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.512769 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml"] Oct 31 23:15:51.536576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.512858 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:15:51.536576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.512903 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:15:51.735604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.724192 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:15:51.815012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.811464 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:51.815532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.812225 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:51.817408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.736733 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:15:51.854133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.818122 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:15:51.854392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.854371 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:15:51.854591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.854578 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:15:51.854714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.854702 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-56b15fa9-5lsml"] Oct 31 23:15:51.899041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.899014 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:15:51.916442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.915956 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv"] Oct 31 23:15:51.999044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:51.998511 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:52.026826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.024152 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:15:52.078677 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:15:52.078179 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda224111b_9baa_4d79_a090_d3eb859b7839.slice/crio-dc9d4d78c24ff0f268ca296f405d0775fe403d8eca5bc44c64f9367319211dae\": RecentStats: unable to find data in memory cache]" Oct 31 23:15:52.133577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.125217 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:15:52.133577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.126102 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:15:52.229482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.229432 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba53702-6261-456b-9e35-790b8ec54e44" path="/var/lib/kubelet/pods/4ba53702-6261-456b-9e35-790b8ec54e44/volumes" Oct 31 23:15:52.244244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.244221 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:15:52.288417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.288386 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:52.455817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.455769 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6"] Oct 31 23:15:52.473855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.473823 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.492079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.492052 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8e6a08-openshift-default-dockercfg-btdj9\"" Oct 31 23:15:52.498217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.498163 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6"] Oct 31 23:15:52.550048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.550014 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568360 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568416 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568448 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568516 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568845 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568889 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6mv8\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568948 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.568992 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.571567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.569107 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.670364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.670322 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671200 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.670444 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671452 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671492 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671595 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671732 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671764 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6mv8\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671861 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671830 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.671918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.671883 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.672265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.672245 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.672799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.672771 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.672992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.672963 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.673324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.673245 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.675807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.675780 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.685045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.685023 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6mv8\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.685885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.685860 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.698915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.698886 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.709220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.709192 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:52.796747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.796658 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:52.861047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.860999 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" event={"ID":"a224111b-9baa-4d79-a090-d3eb859b7839","Type":"ContainerStarted","Data":"40e43c90b5d67053b96b650f6cd994979e2aef0a539c7a2bfadf35669e21f1e3"} Oct 31 23:15:52.861315 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.861284 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" event={"ID":"a224111b-9baa-4d79-a090-d3eb859b7839","Type":"ContainerStarted","Data":"dc9d4d78c24ff0f268ca296f405d0775fe403d8eca5bc44c64f9367319211dae"} Oct 31 23:15:52.906767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:52.906705 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" podStartSLOduration=4.906685221 podStartE2EDuration="4.906685221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:52.892979643 +0000 UTC m=+1579.122534056" watchObservedRunningTime="2026-10-31 23:15:52.906685221 +0000 UTC m=+1579.136239632" Oct 31 23:15:53.012521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.012434 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:53.429467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.426001 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6"] Oct 31 23:15:53.443652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.442908 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:53.443652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.442994 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:53.443652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.443035 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:15:53.869673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.869636 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:53.882836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.882804 5043 generic.go:356] "Generic (PLEG): container finished" podID="a224111b-9baa-4d79-a090-d3eb859b7839" containerID="40e43c90b5d67053b96b650f6cd994979e2aef0a539c7a2bfadf35669e21f1e3" exitCode=0 Oct 31 23:15:53.883164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.883142 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" event={"ID":"a224111b-9baa-4d79-a090-d3eb859b7839","Type":"ContainerDied","Data":"40e43c90b5d67053b96b650f6cd994979e2aef0a539c7a2bfadf35669e21f1e3"} Oct 31 23:15:53.887929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.886402 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" event={"ID":"433ee218-d3e7-4837-98f8-642a6266fdf9","Type":"ContainerStarted","Data":"5d27d729108bc3f61da8492942764f57bdedfe471909d91a2e519b823497c6c0"} Oct 31 23:15:53.887929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.886455 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" event={"ID":"433ee218-d3e7-4837-98f8-642a6266fdf9","Type":"ContainerStarted","Data":"bfca8a6e0741e2582db5d7cca8d179673656826dfcfc2a33a6c70628d4bb47c9"} Oct 31 23:15:53.943803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:53.943718 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podStartSLOduration=1.943677498 podStartE2EDuration="1.943677498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:53.941808939 +0000 UTC m=+1580.171363350" watchObservedRunningTime="2026-10-31 23:15:53.943677498 +0000 UTC m=+1580.173231936" Oct 31 23:15:54.595189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:54.595089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:54.822571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:54.822239 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:54.849617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:54.848375 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:54.924251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:54.923782 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:54.924826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:54.924803 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:15:55.244507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.244478 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:55.275245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.275209 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4\" (UniqueName: \"kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4\") pod \"a224111b-9baa-4d79-a090-d3eb859b7839\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " Oct 31 23:15:55.275421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.275352 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca\") pod \"a224111b-9baa-4d79-a090-d3eb859b7839\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " Oct 31 23:15:55.276082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.276051 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca\") pod \"a224111b-9baa-4d79-a090-d3eb859b7839\" (UID: \"a224111b-9baa-4d79-a090-d3eb859b7839\") " Oct 31 23:15:55.278534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.278506 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca" pod "a224111b-9baa-4d79-a090-d3eb859b7839" (UID: "a224111b-9baa-4d79-a090-d3eb859b7839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:15:55.278781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.278763 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca" pod "a224111b-9baa-4d79-a090-d3eb859b7839" (UID: "a224111b-9baa-4d79-a090-d3eb859b7839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:15:55.279190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.279175 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:55.279255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.279192 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a224111b-9baa-4d79-a090-d3eb859b7839-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:55.281656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.281618 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4" pod "a224111b-9baa-4d79-a090-d3eb859b7839" (UID: "a224111b-9baa-4d79-a090-d3eb859b7839"). InnerVolumeSpecName "kube-api-access-6pwt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:15:55.381406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.381368 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwt4\" (UniqueName: \"kubernetes.io/projected/a224111b-9baa-4d79-a090-d3eb859b7839-kube-api-access-6pwt4\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:15:55.622569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.621095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:55.941620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.941426 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" Oct 31 23:15:55.952224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.952181 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv" event={"ID":"a224111b-9baa-4d79-a090-d3eb859b7839","Type":"ContainerDied","Data":"dc9d4d78c24ff0f268ca296f405d0775fe403d8eca5bc44c64f9367319211dae"} Oct 31 23:15:55.952224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:55.952222 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9d4d78c24ff0f268ca296f405d0775fe403d8eca5bc44c64f9367319211dae" Oct 31 23:15:56.030072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.030033 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv"] Oct 31 23:15:56.036843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.036816 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbca134-5tmrv"] Oct 31 23:15:56.043490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.043464 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w"] Oct 31 23:15:56.053024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.052993 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a9cf17-nbj6w"] Oct 31 23:15:56.306615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.304785 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:56.463701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.463669 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:56.635909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:56.635831 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:57.201758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.199886 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:57.635787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.635748 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:57.674147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.674116 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b"] Oct 31 23:15:57.674961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.674940 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a224111b-9baa-4d79-a090-d3eb859b7839" containerName="revoke-keys" Oct 31 23:15:57.675059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.674962 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a224111b-9baa-4d79-a090-d3eb859b7839" containerName="revoke-keys" Oct 31 23:15:57.675118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.675106 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a224111b-9baa-4d79-a090-d3eb859b7839" containerName="revoke-keys" Oct 31 23:15:57.678612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.678591 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:57.682663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.682638 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8e6a08\"" Oct 31 23:15:57.684149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.684130 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8e6a08-dockercfg-dn8jq\"" Oct 31 23:15:57.690054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.690033 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b"] Oct 31 23:15:57.743663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.743624 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs642\" (UniqueName: \"kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642\") pod \"payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:57.743926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.743902 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:57.833568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.812157 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:57.966710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.927455 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs642\" (UniqueName: \"kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642\") pod \"payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:57.966710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.966342 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:57.967901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:57.967430 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:58.031587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.019298 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs642\" (UniqueName: \"kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642\") pod \"payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:58.130646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.130603 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46bdbaf-4f06-4e52-900b-74f15e5f4c4e" path="/var/lib/kubelet/pods/b46bdbaf-4f06-4e52-900b-74f15e5f4c4e/volumes" Oct 31 23:15:58.131743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.131721 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1982c4-16ae-453d-b617-8e69cead1b3a" path="/var/lib/kubelet/pods/cb1982c4-16ae-453d-b617-8e69cead1b3a/volumes" Oct 31 23:15:58.135024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.134873 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm"] Oct 31 23:15:58.148111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.146883 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5"] Oct 31 23:15:58.158283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.158258 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84657d16-7cxpm"] Oct 31 23:15:58.166344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.166315 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c18ddf6-xdvp5"] Oct 31 23:15:58.296740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.296705 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:58.345568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.344994 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:58.661511 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.661481 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b"] Oct 31 23:15:58.661903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:15:58.661877 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947422e9_fc75_4947_b557_cd04e8f24974.slice/crio-eb77938699c5584367aa542c0b089c75c467803599d7289e34b2ef6801d41f5f WatchSource:0}: Error finding container eb77938699c5584367aa542c0b089c75c467803599d7289e34b2ef6801d41f5f: Status 404 returned error can't find the container with id eb77938699c5584367aa542c0b089c75c467803599d7289e34b2ef6801d41f5f Oct 31 23:15:58.965627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:58.961148 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:59.111030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.110990 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" event={"ID":"947422e9-fc75-4947-b557-cd04e8f24974","Type":"ContainerStarted","Data":"b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b"} Oct 31 23:15:59.111218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.111060 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:15:59.111218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.111083 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" event={"ID":"947422e9-fc75-4947-b557-cd04e8f24974","Type":"ContainerStarted","Data":"eb77938699c5584367aa542c0b089c75c467803599d7289e34b2ef6801d41f5f"} Oct 31 23:15:59.132377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.132069 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j"] Oct 31 23:15:59.154224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.154124 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podStartSLOduration=2.154097008 podStartE2EDuration="2.154097008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:15:59.132962873 +0000 UTC m=+1585.362517285" watchObservedRunningTime="2026-10-31 23:15:59.154097008 +0000 UTC m=+1585.383651419" Oct 31 23:15:59.243951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.243918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:59.418568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.413176 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:59.866679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.866638 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:15:59.968948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.968915 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0499052f-9164-4ef0-b9d1-fc4ee4168900" path="/var/lib/kubelet/pods/0499052f-9164-4ef0-b9d1-fc4ee4168900/volumes" Oct 31 23:15:59.969782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:15:59.969758 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a97f61d-f903-42dc-ba88-ba0d98cd77b5" path="/var/lib/kubelet/pods/1a97f61d-f903-42dc-ba88-ba0d98cd77b5/volumes" Oct 31 23:16:00.055566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:00.053592 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q"] Oct 31 23:16:00.062095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:00.062061 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f0abaa7-9nl5q"] Oct 31 23:16:00.523567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:00.521050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:01.153583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.147414 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:01.320568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.317146 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:01.538571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.532627 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv"] Oct 31 23:16:01.538571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.533107 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:01.565565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.555754 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-907d182f-59snv"] Oct 31 23:16:01.678516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.678473 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:01.956108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.956030 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a0d684-84d4-4e5a-ad73-5e3615604a98" path="/var/lib/kubelet/pods/62a0d684-84d4-4e5a-ad73-5e3615604a98/volumes" Oct 31 23:16:01.956575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:01.956536 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a224111b-9baa-4d79-a090-d3eb859b7839" path="/var/lib/kubelet/pods/a224111b-9baa-4d79-a090-d3eb859b7839/volumes" Oct 31 23:16:02.055034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:02.054996 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2"] Oct 31 23:16:02.061474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:02.061445 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f47faab-ll7p2"] Oct 31 23:16:02.413581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:02.341803 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:02.485277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:02.485243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:02.650885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:02.650849 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:03.104852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:03.104802 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb"] Oct 31 23:16:03.105947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:03.105909 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" podUID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" containerName="istio-proxy" containerID="cri-o://e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217" gracePeriod=30 Oct 31 23:16:03.144843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:03.143692 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:03.290929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:03.290892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:03.913588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:03.913496 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:04.010192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.010160 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3543cacb-852e-4374-840f-03acd54a50e8" path="/var/lib/kubelet/pods/3543cacb-852e-4374-840f-03acd54a50e8/volumes" Oct 31 23:16:04.074226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.074195 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:04.133523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.133484 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" podUID="be07e464-5d0a-4050-8ae7-49f904f56811" containerName="payload-processing" containerID="cri-o://1630c2c3fdbc7510ad1db16422ed42a10aee8600eb28758b124ab096424dffd7" gracePeriod=25 Oct 31 23:16:04.305510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.305479 5043 generic.go:356] "Generic (PLEG): container finished" podID="be07e464-5d0a-4050-8ae7-49f904f56811" containerID="1630c2c3fdbc7510ad1db16422ed42a10aee8600eb28758b124ab096424dffd7" exitCode=0 Oct 31 23:16:04.305774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.305699 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" event={"ID":"be07e464-5d0a-4050-8ae7-49f904f56811","Type":"ContainerDied","Data":"1630c2c3fdbc7510ad1db16422ed42a10aee8600eb28758b124ab096424dffd7"} Oct 31 23:16:04.458606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.458384 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:04.789570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.786865 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:16:04.907583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:04.869180 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:05.051338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.050859 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp\" (UniqueName: \"kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp\") pod \"be07e464-5d0a-4050-8ae7-49f904f56811\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " Oct 31 23:16:05.051338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.051058 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume\") pod \"be07e464-5d0a-4050-8ae7-49f904f56811\" (UID: \"be07e464-5d0a-4050-8ae7-49f904f56811\") " Oct 31 23:16:05.052249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.052218 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume" pod "be07e464-5d0a-4050-8ae7-49f904f56811" (UID: "be07e464-5d0a-4050-8ae7-49f904f56811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:16:05.055490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.055463 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp" pod "be07e464-5d0a-4050-8ae7-49f904f56811" (UID: "be07e464-5d0a-4050-8ae7-49f904f56811"). InnerVolumeSpecName "kube-api-access-4pgdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:16:05.176647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.156993 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be07e464-5d0a-4050-8ae7-49f904f56811-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:05.176939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.176917 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgdp\" (UniqueName: \"kubernetes.io/projected/be07e464-5d0a-4050-8ae7-49f904f56811-kube-api-access-4pgdp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:05.177033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.161854 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:05.410635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.396505 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" event={"ID":"be07e464-5d0a-4050-8ae7-49f904f56811","Type":"ContainerDied","Data":"e95e86abfdb76e3b4c296cffbfc505f1551a875b061cb8faca0c13eb2c619ae3"} Oct 31 23:16:05.410635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.396665 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j" Oct 31 23:16:05.431571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.429000 5043 scope.go:122] "RemoveContainer" containerID="1630c2c3fdbc7510ad1db16422ed42a10aee8600eb28758b124ab096424dffd7" Oct 31 23:16:05.587422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.587388 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j"] Oct 31 23:16:05.593642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.593310 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-907d182f-648bf98f76-hb76j"] Oct 31 23:16:05.689426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.688840 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:05.966445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:05.966366 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be07e464-5d0a-4050-8ae7-49f904f56811" path="/var/lib/kubelet/pods/be07e464-5d0a-4050-8ae7-49f904f56811/volumes" Oct 31 23:16:06.454577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:06.452089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:06.659926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:06.659892 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:07.223830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:07.223739 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:07.919604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:07.918438 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:08.071734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:08.071699 5043 patch_prober.go:29] interesting pod/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:16:08.072021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:08.071988 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" podUID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:16:08.612081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:08.611904 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:09.033759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:09.033725 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck"] Oct 31 23:16:09.040234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:09.040208 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd82abeb-2ccck"] Oct 31 23:16:09.121707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:09.121673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:09.255323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:09.255289 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:09.605012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:09.604979 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:09.942258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:09.942179 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bfe778-b8a8-4309-b30e-03c962094983" path="/var/lib/kubelet/pods/43bfe778-b8a8-4309-b30e-03c962094983/volumes" Oct 31 23:16:10.157753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.157726 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:16:10.172782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.172754 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk"] Oct 31 23:16:10.179077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.179048 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh"] Oct 31 23:16:10.186496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.186471 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcadf2c-26cgh"] Oct 31 23:16:10.192336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.192279 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c95ca12-4cxsk"] Oct 31 23:16:10.446322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.446243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:10.476095 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:16:10.476028 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:16:10.508912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.508882 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4"] Oct 31 23:16:10.509660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.509642 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be07e464-5d0a-4050-8ae7-49f904f56811" containerName="payload-processing" Oct 31 23:16:10.509764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.509662 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be07e464-5d0a-4050-8ae7-49f904f56811" containerName="payload-processing" Oct 31 23:16:10.509817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.509808 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be07e464-5d0a-4050-8ae7-49f904f56811" containerName="payload-processing" Oct 31 23:16:10.608324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.608290 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4"] Oct 31 23:16:10.619529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.609942 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:10.657183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.657133 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:10.698714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.698560 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:10.698714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.691419 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wn9\" (UniqueName: \"kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9\") pod \"payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:10.709187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.705704 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f80dfa-dockercfg-9zjwd\"" Oct 31 23:16:10.709187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.705826 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f80dfa\"" Oct 31 23:16:10.830234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.819006 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:10.830234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.819162 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wn9\" (UniqueName: \"kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9\") pod \"payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:10.830234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.829069 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:10.848572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:10.847050 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wn9\" (UniqueName: \"kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9\") pod \"payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:11.119646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:11.119616 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:11.536571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:11.533413 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:11.592276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:11.592238 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4"] Oct 31 23:16:11.608001 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:16:11.607954 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf852be0e_4036_4565_a1c1_d20c99495370.slice/crio-cf89b18638723a639d5415fdedc8232dd38b3c25e39b2a84fcb66eeb81e2f79c WatchSource:0}: Error finding container cf89b18638723a639d5415fdedc8232dd38b3c25e39b2a84fcb66eeb81e2f79c: Status 404 returned error can't find the container with id cf89b18638723a639d5415fdedc8232dd38b3c25e39b2a84fcb66eeb81e2f79c Oct 31 23:16:11.955128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:11.950579 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0262ee16-d19e-4edc-90a2-393a65efca41" path="/var/lib/kubelet/pods/0262ee16-d19e-4edc-90a2-393a65efca41/volumes" Oct 31 23:16:11.955128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:11.951313 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28c5f7a-39f4-492d-b560-09f385366c29" path="/var/lib/kubelet/pods/d28c5f7a-39f4-492d-b560-09f385366c29/volumes" Oct 31 23:16:12.055525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.055492 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s"] Oct 31 23:16:12.066404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.066375 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ae13b5-56g9s"] Oct 31 23:16:12.407850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.407815 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:12.510450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.510410 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" event={"ID":"f852be0e-4036-4565-a1c1-d20c99495370","Type":"ContainerStarted","Data":"73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d"} Oct 31 23:16:12.510664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.510485 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:12.510664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.510511 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" event={"ID":"f852be0e-4036-4565-a1c1-d20c99495370","Type":"ContainerStarted","Data":"cf89b18638723a639d5415fdedc8232dd38b3c25e39b2a84fcb66eeb81e2f79c"} Oct 31 23:16:12.534873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:12.534826 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podStartSLOduration=2.534813432 podStartE2EDuration="2.534813432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:12.532664635 +0000 UTC m=+1598.762219045" watchObservedRunningTime="2026-10-31 23:16:12.534813432 +0000 UTC m=+1598.764367864" Oct 31 23:16:13.026556 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:13.026510 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd"] Oct 31 23:16:13.032792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:13.032762 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ace810-blzdd"] Oct 31 23:16:13.134771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:13.134739 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:13.782228 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:13.782179 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:13.938616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:13.938537 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20deaa91-4004-4e33-be85-a3cb7279b160" path="/var/lib/kubelet/pods/20deaa91-4004-4e33-be85-a3cb7279b160/volumes" Oct 31 23:16:13.939363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:13.939204 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d81d6c-ce05-4db7-a3dd-3724fd75a874" path="/var/lib/kubelet/pods/63d81d6c-ce05-4db7-a3dd-3724fd75a874/volumes" Oct 31 23:16:14.445477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:14.444111 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:15.038114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:15.038076 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:15.193422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:15.193380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:15.886326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:15.886287 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:16.588810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:16.588770 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:17.178961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.178926 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:16:17.363693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.363145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367040 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367200 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367335 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367374 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367515 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367631 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367665 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367779 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.368571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.367919 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket\") pod \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\" (UID: \"ac53ef9e-c3b9-41c6-aa58-dcbce759111e\") " Oct 31 23:16:17.370269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.370232 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:16:17.370858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.370657 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:16:17.371010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.370919 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:16:17.371178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.371159 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:16:17.371283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.371226 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:16:17.375920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.375756 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:16:17.376522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.376450 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "kube-api-access-5cbv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:16:17.378484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.378458 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:16:17.381003 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.379044 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo" pod "ac53ef9e-c3b9-41c6-aa58-dcbce759111e" (UID: "ac53ef9e-c3b9-41c6-aa58-dcbce759111e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:16:17.470312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470279 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470309 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470319 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470328 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470339 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470349 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbv8\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-kube-api-access-5cbv8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470359 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470368 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.470607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.470376 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac53ef9e-c3b9-41c6-aa58-dcbce759111e-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:17.571196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.571163 5043 generic.go:356] "Generic (PLEG): container finished" podID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" containerID="e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217" exitCode=0 Oct 31 23:16:17.571478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.571223 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" Oct 31 23:16:17.571478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.571250 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" event={"ID":"ac53ef9e-c3b9-41c6-aa58-dcbce759111e","Type":"ContainerDied","Data":"e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217"} Oct 31 23:16:17.571478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.571321 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb" event={"ID":"ac53ef9e-c3b9-41c6-aa58-dcbce759111e","Type":"ContainerDied","Data":"80d9ec767897f6586b3a95d0fe067380b0cd96aa578c593bf2270f14c73abc23"} Oct 31 23:16:17.571478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.571349 5043 scope.go:122] "RemoveContainer" containerID="e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217" Oct 31 23:16:17.585164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.585146 5043 scope.go:122] "RemoveContainer" containerID="e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217" Oct 31 23:16:17.585437 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:16:17.585417 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217\": container with ID starting with e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217 not found: ID does not exist" containerID="e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217" Oct 31 23:16:17.585485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.585446 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217"} err="failed to get container status \"e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217\": rpc error: code = NotFound desc = could not find container \"e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217\": container with ID starting with e7580db73b13333da0425e1e58fb7dfd65f9b476338ce96c8505366ffdb6f217 not found: ID does not exist" Oct 31 23:16:17.600861 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.600841 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb"] Oct 31 23:16:17.606225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:17.606203 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-907d182f-openshift-default-7d69697b6c-nhdsb"] Oct 31 23:16:18.008576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:18.005113 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" path="/var/lib/kubelet/pods/ac53ef9e-c3b9-41c6-aa58-dcbce759111e/volumes" Oct 31 23:16:18.294289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:18.294208 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:19.110466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:19.110428 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:19.826569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:19.819061 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:20.418701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:20.413185 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:21.098615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:21.091021 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:21.578845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:21.578807 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:21.752791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:21.752760 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:22.028587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.028534 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:22.421920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.421830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:22.428220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.428193 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8"] Oct 31 23:16:22.429454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.429425 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" containerName="istio-proxy" Oct 31 23:16:22.429454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.429453 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" containerName="istio-proxy" Oct 31 23:16:22.429736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.429710 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac53ef9e-c3b9-41c6-aa58-dcbce759111e" containerName="istio-proxy" Oct 31 23:16:22.437165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.437141 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.443770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.443743 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8"] Oct 31 23:16:22.443947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.443930 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-d6685e-openshift-default-dockercfg-lz8tn\"" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.526854 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.526936 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527092 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527137 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527186 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527224 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527305 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527334 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbkz\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.535568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.527420 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.631764 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.631839 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.631889 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.631927 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.632002 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.632031 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbkz\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.632114 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.632255 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.633572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.632304 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.659569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.636926 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.659569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.637210 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.659569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.637777 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.659569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.638427 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.659569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.638701 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.693618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.676304 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:22.694600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.694508 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbkz\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.754568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.711172 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.754568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.731017 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.771567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.755093 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy\") pod \"e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:22.771567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:22.766855 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:23.013494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.013133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:23.211279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.211201 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:23.211386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.211340 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:23.211426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.211407 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:23.234827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.233304 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:23.245523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.245484 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8"] Oct 31 23:16:23.527069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.527036 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:16:23.765402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.765357 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" event={"ID":"5f5396c0-d04b-4150-9ad9-179d7319cdb4","Type":"ContainerStarted","Data":"b906f09b982eaecdf92a12fafa3a52c67ac50d1c190b91475c330fa7890d54e0"} Oct 31 23:16:23.765584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.765426 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" event={"ID":"5f5396c0-d04b-4150-9ad9-179d7319cdb4","Type":"ContainerStarted","Data":"0acd768b08236c1bb7c451642e7dc8ca0f4a2cfd08a0beea03fbae06f9e3d46c"} Oct 31 23:16:23.795449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.795363 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" podStartSLOduration=1.795349908 podStartE2EDuration="1.795349908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:23.789315913 +0000 UTC m=+1610.018870344" watchObservedRunningTime="2026-10-31 23:16:23.795349908 +0000 UTC m=+1610.024904317" Oct 31 23:16:23.810103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.810076 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:23.988616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:23.981213 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:24.630570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:24.620529 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:24.815822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:24.815792 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:24.844691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:24.844660 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:25.030054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:25.030023 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf"] Oct 31 23:16:25.039262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:25.039228 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee75fd08-dgrlf"] Oct 31 23:16:25.496601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:25.496566 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:25.785222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:25.785136 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:25.786079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:25.786050 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:16:25.939098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:25.939009 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a237a2-25ab-43b9-a748-359a585a6b0e" path="/var/lib/kubelet/pods/a8a237a2-25ab-43b9-a748-359a585a6b0e/volumes" Oct 31 23:16:26.259568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:26.257981 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:26.420129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:26.420087 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:26.683566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:26.683459 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:27.581813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:27.581765 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:27.742601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:27.742536 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:28.074318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:28.074282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:28.546465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:28.546430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:29.146773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:29.146739 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:30.021900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:30.021762 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:30.697533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:30.697480 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:31.103813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.103779 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng"] Oct 31 23:16:31.149912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.149881 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.177655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.166313 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:16:31.177655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.168722 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:16:31.177655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.168937 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:16:31.177655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.169037 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:16:31.188766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.188739 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng"] Oct 31 23:16:31.310291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.310250 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqck\" (UniqueName: \"kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.310737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.310705 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.311021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.311002 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.421837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.421743 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.423054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.423029 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqck\" (UniqueName: \"kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.424113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.424090 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.424819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.422764 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.467468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.467432 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.492567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.473434 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqck\" (UniqueName: \"kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck\") pod \"maas-api-revoke-sub-8ac0c695-qgbng\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.513992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.513943 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:31.602244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:31.602193 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:32.077586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.075640 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng"] Oct 31 23:16:32.080347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.080307 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8"] Oct 31 23:16:32.086492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.086468 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88"] Oct 31 23:16:32.092993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.092973 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114db158-8qlj8"] Oct 31 23:16:32.099839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.099477 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0497054-hlb88"] Oct 31 23:16:32.320256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.320082 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:32.897795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.897697 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" event={"ID":"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e","Type":"ContainerStarted","Data":"a60a1294668bd07f0b4a9b059700cc3f5f5797556cf6765066b3ece5bdd064b2"} Oct 31 23:16:32.897795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.897768 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" event={"ID":"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e","Type":"ContainerStarted","Data":"511c1ffe9cfc249811bad4b3bf3b8132597df13383c1b3049d0883fa94fa731b"} Oct 31 23:16:32.934571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:32.934478 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" podStartSLOduration=1.9344578430000001 podStartE2EDuration="1.934457843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:32.917784138 +0000 UTC m=+1619.147338574" watchObservedRunningTime="2026-10-31 23:16:32.934457843 +0000 UTC m=+1619.164012257" Oct 31 23:16:33.057805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:33.057769 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:33.828134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:33.828075 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:33.993863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:33.993817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:34.007622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.007582 5043 generic.go:356] "Generic (PLEG): container finished" podID="f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" containerID="a60a1294668bd07f0b4a9b059700cc3f5f5797556cf6765066b3ece5bdd064b2" exitCode=0 Oct 31 23:16:34.033565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.031605 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27be5640-28a4-4950-856a-628e861d09fb" path="/var/lib/kubelet/pods/27be5640-28a4-4950-856a-628e861d09fb/volumes" Oct 31 23:16:34.056603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.047853 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbdbff0-b677-4495-95a6-f5884856bc4a" path="/var/lib/kubelet/pods/2fbdbff0-b677-4495-95a6-f5884856bc4a/volumes" Oct 31 23:16:34.056603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.048245 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" event={"ID":"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e","Type":"ContainerDied","Data":"a60a1294668bd07f0b4a9b059700cc3f5f5797556cf6765066b3ece5bdd064b2"} Oct 31 23:16:34.303949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.303916 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:34.697736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.697630 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:34.833064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:34.833028 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:35.388581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.369288 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:35.654576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.649633 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:35.655836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.655802 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:35.759572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.759281 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca\") pod \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " Oct 31 23:16:35.759572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.759488 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca\") pod \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " Oct 31 23:16:35.759906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.759619 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck\" (UniqueName: \"kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck\") pod \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\" (UID: \"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e\") " Oct 31 23:16:35.762634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.761211 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca" pod "f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" (UID: "f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:16:35.764470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.764443 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca" pod "f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" (UID: "f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:16:35.767958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.767053 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck" pod "f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" (UID: "f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e"). InnerVolumeSpecName "kube-api-access-vfqck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:16:35.825817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.824767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:35.862815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.862783 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:35.862815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.862813 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:35.862815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:35.862824 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqck\" (UniqueName: \"kubernetes.io/projected/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e-kube-api-access-vfqck\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:16:36.037231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:36.037191 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" Oct 31 23:16:36.037422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:36.037375 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng" event={"ID":"f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e","Type":"ContainerDied","Data":"511c1ffe9cfc249811bad4b3bf3b8132597df13383c1b3049d0883fa94fa731b"} Oct 31 23:16:36.037479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:36.037457 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="511c1ffe9cfc249811bad4b3bf3b8132597df13383c1b3049d0883fa94fa731b" Oct 31 23:16:36.230337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:36.230249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:36.918661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:36.918617 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:37.067905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.067871 5043 scope.go:122] "RemoveContainer" containerID="ebee9ac71b5803c68c4e4128fb60cd6f16459cc22ae9eb8a19a66075576fbc46" Oct 31 23:16:37.561567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.560743 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs"] Oct 31 23:16:37.561875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.561854 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" containerName="revoke-keys" Oct 31 23:16:37.561875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.561875 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" containerName="revoke-keys" Oct 31 23:16:37.562069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.562058 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" containerName="revoke-keys" Oct 31 23:16:37.594181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.594147 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs"] Oct 31 23:16:37.594181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.594176 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.624146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.624114 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:37.708444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.708411 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.709381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.709345 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.709601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.709574 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.709766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.709745 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.709866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.709844 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.711051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.711002 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.711051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.711048 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.711291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.711086 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xfx\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.711291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.711117 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.812989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.812909 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.812989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.812960 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813121 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813212 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813264 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813297 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813348 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xfx\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813384 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813512 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813533 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813592 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.813872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.813858 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.814185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.814160 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.816108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.816084 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.816225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.816106 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.827158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.827134 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.827720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.827700 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xfx\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx\") pod \"e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:37.922335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:37.922303 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:38.293985 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:16:38.293951 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce8af1c_a77d_44b2_920d_9d57b20a21ad.slice/crio-85ac2c632c6e385ead346ff34dd220198cd20b0b1bddcfa813a1079662409ffa WatchSource:0}: Error finding container 85ac2c632c6e385ead346ff34dd220198cd20b0b1bddcfa813a1079662409ffa: Status 404 returned error can't find the container with id 85ac2c632c6e385ead346ff34dd220198cd20b0b1bddcfa813a1079662409ffa Oct 31 23:16:38.297102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:38.297067 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:38.297184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:38.297136 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:38.297184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:38.297163 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:38.299699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:38.299673 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs"] Oct 31 23:16:38.423009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:38.422826 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:38.868007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:38.867979 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:39.061976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.061945 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:39.234459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.086628 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" event={"ID":"4ce8af1c-a77d-44b2-920d-9d57b20a21ad","Type":"ContainerStarted","Data":"803b0c505a846b722a9c109dd0336a41787b7113073da1059a1dcfd4d5f95870"} Oct 31 23:16:39.234459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.086690 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" event={"ID":"4ce8af1c-a77d-44b2-920d-9d57b20a21ad","Type":"ContainerStarted","Data":"85ac2c632c6e385ead346ff34dd220198cd20b0b1bddcfa813a1079662409ffa"} Oct 31 23:16:39.234459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.152446 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podStartSLOduration=2.152428762 podStartE2EDuration="2.152428762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:39.147221615 +0000 UTC m=+1625.376776040" watchObservedRunningTime="2026-10-31 23:16:39.152428762 +0000 UTC m=+1625.381983172" Oct 31 23:16:39.644591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.642387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:39.928815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.928239 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:39.928815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.928359 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 23:16:39.928815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.928401 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 23:16:39.936699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.936580 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:39.936699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:39.936636 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:40.083566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.083521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:40.092757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.092724 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz"] Oct 31 23:16:40.098652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.098629 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.107889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.107860 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-7c4c91\"" Oct 31 23:16:40.109514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.108898 5043 generic.go:356] "Generic (PLEG): container finished" podID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerID="5d27d729108bc3f61da8492942764f57bdedfe471909d91a2e519b823497c6c0" exitCode=0 Oct 31 23:16:40.109514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.109458 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" event={"ID":"433ee218-d3e7-4837-98f8-642a6266fdf9","Type":"ContainerDied","Data":"5d27d729108bc3f61da8492942764f57bdedfe471909d91a2e519b823497c6c0"} Oct 31 23:16:40.110352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.110174 5043 scope.go:122] "RemoveContainer" containerID="5d27d729108bc3f61da8492942764f57bdedfe471909d91a2e519b823497c6c0" Oct 31 23:16:40.112407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.112372 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:40.112513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.112459 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:40.112513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.112506 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:16:40.146567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.124241 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz"] Oct 31 23:16:40.184621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.182956 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.184621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.183016 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bc9\" (UniqueName: \"kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9\") pod \"payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.211833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.209036 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4"] Oct 31 23:16:40.531597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.514754 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:40.533203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.501074 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:40.703051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.703019 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4"] Oct 31 23:16:40.729100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.729063 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.729374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.729350 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bc9\" (UniqueName: \"kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9\") pod \"payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.731694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.731669 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.754748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.753911 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-7c4c91-dockercfg-t222n\"" Oct 31 23:16:40.763714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.760311 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bc9\" (UniqueName: \"kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9\") pod \"payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.782126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.782067 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:40.790877 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:16:40.790844 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:16:40.860111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.860070 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:40.860462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.860437 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnl9s\" (UniqueName: \"kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s\") pod \"payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:40.890571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.887502 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:40.926399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.925960 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 23:16:40.926399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.926023 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 23:16:40.964571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.964093 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:40.964571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.964202 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnl9s\" (UniqueName: \"kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s\") pod \"payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:40.985752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.985720 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:40.988593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:40.987225 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnl9s\" (UniqueName: \"kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s\") pod \"payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:41.105182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.105152 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:41.205565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.204627 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" event={"ID":"433ee218-d3e7-4837-98f8-642a6266fdf9","Type":"ContainerStarted","Data":"96003ef535034fb9211ba89a8bbf1708a2d9944b47b65cc9bf52e3ab97683493"} Oct 31 23:16:41.281781 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:16:41.281012 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7b510a_9644_48e0_96fc_29447a106ee5.slice/crio-19cd6f8f3978f70d45cea83ec04bfa1e18676ecf7caa2a5a68e033e37834e131 WatchSource:0}: Error finding container 19cd6f8f3978f70d45cea83ec04bfa1e18676ecf7caa2a5a68e033e37834e131: Status 404 returned error can't find the container with id 19cd6f8f3978f70d45cea83ec04bfa1e18676ecf7caa2a5a68e033e37834e131 Oct 31 23:16:41.298159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.297882 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz"] Oct 31 23:16:41.376671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.375159 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:41.671750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.671673 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:41.688923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.674322 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4"] Oct 31 23:16:41.746660 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:16:41.746596 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552eb1a2_3e7e_4311_add3_9dba9f0416af.slice/crio-a745ac0d0a7eddfff1a7f43daa6a084cf1f05de7ef4775a4ba3ef54285d7dbb0 WatchSource:0}: Error finding container a745ac0d0a7eddfff1a7f43daa6a084cf1f05de7ef4775a4ba3ef54285d7dbb0: Status 404 returned error can't find the container with id a745ac0d0a7eddfff1a7f43daa6a084cf1f05de7ef4775a4ba3ef54285d7dbb0 Oct 31 23:16:41.797801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.797409 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:16:41.798341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.798035 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:41.798341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.798083 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:41.931583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.928706 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 23:16:41.931583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:41.928765 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 23:16:42.236623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.236466 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" event={"ID":"552eb1a2-3e7e-4311-add3-9dba9f0416af","Type":"ContainerStarted","Data":"23fca3cc4cc839371091a211e909edffc0770eba7d2c7d79b9d1d02b695df0eb"} Oct 31 23:16:42.236623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.236533 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" event={"ID":"552eb1a2-3e7e-4311-add3-9dba9f0416af","Type":"ContainerStarted","Data":"a745ac0d0a7eddfff1a7f43daa6a084cf1f05de7ef4775a4ba3ef54285d7dbb0"} Oct 31 23:16:42.265646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.256846 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:42.287600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.281147 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" event={"ID":"da7b510a-9644-48e0-96fc-29447a106ee5","Type":"ContainerStarted","Data":"066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85"} Oct 31 23:16:42.287600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.281212 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" event={"ID":"da7b510a-9644-48e0-96fc-29447a106ee5","Type":"ContainerStarted","Data":"19cd6f8f3978f70d45cea83ec04bfa1e18676ecf7caa2a5a68e033e37834e131"} Oct 31 23:16:42.298972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.298827 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" podStartSLOduration=2.2988158260000002 podStartE2EDuration="2.298815826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:42.295278987 +0000 UTC m=+1628.524833400" watchObservedRunningTime="2026-10-31 23:16:42.298815826 +0000 UTC m=+1628.528370235" Oct 31 23:16:42.380749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.374893 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" podStartSLOduration=2.374869259 podStartE2EDuration="2.374869259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:42.328094429 +0000 UTC m=+1628.557648901" watchObservedRunningTime="2026-10-31 23:16:42.374869259 +0000 UTC m=+1628.604423669" Oct 31 23:16:42.499594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.499421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:42.797959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.797868 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:42.797959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.797926 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:42.935572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.934802 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 23:16:42.935572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:42.934860 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 23:16:43.172269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:43.165148 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:43.383378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:43.383340 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:43.801058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:43.801018 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:43.801402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:43.801368 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:43.977581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:43.976952 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 23:16:43.977581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:43.977015 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 23:16:44.170969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:44.168942 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:44.797302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:44.797264 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:44.797731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:44.797331 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:44.904307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:44.904233 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:44.922996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:44.922958 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 23:16:44.923244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:44.923215 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 23:16:45.592679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:45.592635 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:45.797792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:45.797757 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:45.798282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:45.797809 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:45.959061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:45.958979 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:45.960077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:45.959504 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:45.961388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:45.961366 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:16:46.074809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:46.074776 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj"] Oct 31 23:16:46.081829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:46.081804 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062f13f9-znflj"] Oct 31 23:16:46.258735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:46.258702 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:46.450743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:46.449878 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:46.797127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:46.797086 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 23:16:46.797302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:46.797175 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 23:16:47.061449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.061405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:47.739863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.739816 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:47.811056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.811021 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:16:47.812246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.812221 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:16:47.813104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.813076 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:16:47.907623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.907583 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:47.939318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:47.939282 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836a5652-b050-408b-9050-bd651a649dff" path="/var/lib/kubelet/pods/836a5652-b050-408b-9050-bd651a649dff/volumes" Oct 31 23:16:48.427510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:48.427453 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:48.735534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:48.735451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:48.875624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:48.875589 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:49.340145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:49.340110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:49.568566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:49.567960 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:49.759046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:49.759008 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:50.029512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:50.029437 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54"] Oct 31 23:16:50.038793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:50.038761 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040c1f88-c6j54"] Oct 31 23:16:50.340467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:50.340351 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:51.025239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:51.024709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:51.716018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:51.715984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:51.942871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:51.942835 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb18ce55-68de-4614-8dcc-ecc80ab72dc8" path="/var/lib/kubelet/pods/bb18ce55-68de-4614-8dcc-ecc80ab72dc8/volumes" Oct 31 23:16:52.365615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:52.365532 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:53.215018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:53.214971 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:53.455913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:53.455865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:53.602438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:53.602405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:54.023487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.023447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:54.408105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.408032 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:16:54.408105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.408102 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:16:54.668969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.668863 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:54.713303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.713275 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g"] Oct 31 23:16:54.718252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.718226 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:54.725832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.725802 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-36a15fc5-dockercfg-25gns\"" Oct 31 23:16:54.743493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.743464 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g"] Oct 31 23:16:54.756651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.756622 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq"] Oct 31 23:16:54.762737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.762710 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:54.772623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.772598 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq"] Oct 31 23:16:54.779306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.779286 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-36a15fc5\"" Oct 31 23:16:54.802155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.802103 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:54.802472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.802437 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk478\" (UniqueName: \"kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478\") pod \"payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:54.803395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.802502 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:54.803811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.803784 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xvg6\" (UniqueName: \"kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6\") pod \"payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:54.829224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.829190 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:54.922226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.922134 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:54.923530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.915839 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:54.923886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.923862 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk478\" (UniqueName: \"kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478\") pod \"payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:54.924034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.924017 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:54.924234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.924217 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xvg6\" (UniqueName: \"kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6\") pod \"payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:54.925828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.925806 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:54.992155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:54.986616 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk478\" (UniqueName: \"kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478\") pod \"payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:55.044581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.028316 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xvg6\" (UniqueName: \"kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6\") pod \"payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:55.048571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.044886 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:55.083700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.082861 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:55.543718 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:16:55.543688 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd046db04_b4c5_4213_9bbf_5b6728d7a6f1.slice/crio-9a167ab750fc1d390bd452e6ea204d5f37db724ec6f7c945776de2ee5bcab1b9 WatchSource:0}: Error finding container 9a167ab750fc1d390bd452e6ea204d5f37db724ec6f7c945776de2ee5bcab1b9: Status 404 returned error can't find the container with id 9a167ab750fc1d390bd452e6ea204d5f37db724ec6f7c945776de2ee5bcab1b9 Oct 31 23:16:55.547116 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:16:55.547087 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263eeccd_41ea_40ad_aa08_d8470d5885d4.slice/crio-26ceb855e73c4a356e7850cdb0770e4fca34194d7cfc5f1237f6d9c5a80aad60 WatchSource:0}: Error finding container 26ceb855e73c4a356e7850cdb0770e4fca34194d7cfc5f1237f6d9c5a80aad60: Status 404 returned error can't find the container with id 26ceb855e73c4a356e7850cdb0770e4fca34194d7cfc5f1237f6d9c5a80aad60 Oct 31 23:16:55.550685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.550621 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq"] Oct 31 23:16:55.557839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.557813 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g"] Oct 31 23:16:55.589851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.589800 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:55.663018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.662988 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" event={"ID":"263eeccd-41ea-40ad-aa08-d8470d5885d4","Type":"ContainerStarted","Data":"26ceb855e73c4a356e7850cdb0770e4fca34194d7cfc5f1237f6d9c5a80aad60"} Oct 31 23:16:55.665084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:55.664858 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" event={"ID":"d046db04-b4c5-4213-9bbf-5b6728d7a6f1","Type":"ContainerStarted","Data":"9a167ab750fc1d390bd452e6ea204d5f37db724ec6f7c945776de2ee5bcab1b9"} Oct 31 23:16:56.340358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.340321 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:56.676665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.676583 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" event={"ID":"d046db04-b4c5-4213-9bbf-5b6728d7a6f1","Type":"ContainerStarted","Data":"d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b"} Oct 31 23:16:56.676665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.676662 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:16:56.678633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.678608 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" event={"ID":"263eeccd-41ea-40ad-aa08-d8470d5885d4","Type":"ContainerStarted","Data":"7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2"} Oct 31 23:16:56.678774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.678735 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:16:56.718716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.718658 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" podStartSLOduration=2.718643969 podStartE2EDuration="2.718643969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:56.711818683 +0000 UTC m=+1642.941373095" watchObservedRunningTime="2026-10-31 23:16:56.718643969 +0000 UTC m=+1642.948198380" Oct 31 23:16:56.739718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:56.739671 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podStartSLOduration=2.739658672 podStartE2EDuration="2.739658672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:16:56.733258207 +0000 UTC m=+1642.962812617" watchObservedRunningTime="2026-10-31 23:16:56.739658672 +0000 UTC m=+1642.969213082" Oct 31 23:16:57.155503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:57.145049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:57.598950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:57.598910 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:57.798571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:57.788778 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:58.055089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:58.055053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:58.759567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:58.752906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:58.982404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:58.982363 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:16:59.481503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:16:59.481462 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:00.132093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:00.132035 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:00.787309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:00.787274 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:01.412782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:01.412746 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:01.561176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:01.561129 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:02.240200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:02.239895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:03.334475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:03.334439 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:04.274591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:04.272980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:04.441216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:04.441100 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:04.938823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:04.938320 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk"] Oct 31 23:17:04.995523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:04.995491 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk"] Oct 31 23:17:04.997523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:04.995887 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.017491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.017461 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c4d32e-openshift-default-dockercfg-m7l28\"" Oct 31 23:17:05.070759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.070702 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.071462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.071437 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.071641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.071624 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcqb\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.071830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.071816 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.071997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.071982 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.072212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.072177 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.072779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.072760 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.072944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.072929 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.073092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.073079 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.198776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.165150 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:05.308570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.257106 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.308570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.301266 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.308570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.301309 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcqb\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.308570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.301446 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.308570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.257521 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.308570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.302351 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.332239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.318362 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.332239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.318516 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.332239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.318574 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.332239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.318642 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.332239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.318744 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.349214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.349170 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.349814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.349790 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.352347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.352046 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.360742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.355036 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.363483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.363433 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.397001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.396927 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.412064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.411971 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcqb\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb\") pod \"e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:05.481214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.481116 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:05.650047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.650011 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:05.661914 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:05.661871 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:06.306567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:06.302626 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:17:06.306858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:06.306825 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:17:06.307024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:06.307008 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:17:06.357603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:06.353979 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk"] Oct 31 23:17:06.390675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:06.386753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:07.341572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.313593 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" event={"ID":"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef","Type":"ContainerStarted","Data":"5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211"} Oct 31 23:17:07.342176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.342142 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" event={"ID":"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef","Type":"ContainerStarted","Data":"a5bec2a62c23d00f6d2ade32dd7a51cc01f8da18b644c4d3a468723abd01fef8"} Oct 31 23:17:07.401117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.401052 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podStartSLOduration=3.401030208 podStartE2EDuration="3.401030208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:07.370526759 +0000 UTC m=+1653.600081168" watchObservedRunningTime="2026-10-31 23:17:07.401030208 +0000 UTC m=+1653.630584620" Oct 31 23:17:07.465844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.465797 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:07.722220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.714867 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:07.817231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.817198 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:17:07.844371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.844342 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:17:07.955575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:07.952343 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:17:08.018715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.008907 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": context deadline exceeded" Oct 31 23:17:08.108066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.108026 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:17:08.138329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.133104 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:17:08.303026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.133154 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:08.303907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.240658 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:17:08.304267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.242370 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:08.304640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.279665 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:08.327567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.325796 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:08.327567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.326445 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:17:08.420568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.417949 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:08.686980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.686873 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:08.954616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.953660 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:17:08.954616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:08.954410 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:17:09.034600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.034573 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:17:09.168571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.143626 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:09.168571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.144213 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:17:09.212249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.211382 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:17:09.305686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.302867 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:09.305686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.303411 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:17:09.311222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.310513 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:09.311222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.310834 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:17:09.311808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.311082 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:17:09.314514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.312235 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:17:09.314514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.312291 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:17:09.314514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.312329 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:17:09.328655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.328626 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:09.340574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.329034 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:17:09.352570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.349964 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:17:09.359923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.359788 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:09.396391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.396289 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:17:09.411982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.411956 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:17:09.420634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.420607 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:17:09.632566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.632021 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:17:09.696035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.686088 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:09.696035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.687663 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:09.748293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.746380 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:17:09.884317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:09.884228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:10.073846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:10.073810 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:10.467103 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:17:10.467069 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:17:10.608571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:10.607962 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:11.062330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.062278 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w"] Oct 31 23:17:11.071386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.071353 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f880724d-4pc4w"] Oct 31 23:17:11.124737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.124701 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:11.169344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.169314 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6"] Oct 31 23:17:11.174617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.174593 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.202627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.192848 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c4d32e-dockercfg-d82l9\"" Oct 31 23:17:11.202627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.193513 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c4d32e\"" Oct 31 23:17:11.202627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.193526 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6"] Oct 31 23:17:11.275318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.275280 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6"] Oct 31 23:17:11.288162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.288135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:11.295908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.295727 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6"] Oct 31 23:17:11.296157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.295867 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.352499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.352193 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.352499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.352267 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgn9\" (UniqueName: \"kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9\") pod \"payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.471593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.459065 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.471593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.459148 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgn9\" (UniqueName: \"kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9\") pod \"payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.471593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.459221 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.471593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.459375 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd276\" (UniqueName: \"kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276\") pod \"payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.471593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.460442 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.622796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.622710 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.624207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.624179 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd276\" (UniqueName: \"kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276\") pod \"payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.637521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.637486 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgn9\" (UniqueName: \"kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9\") pod \"payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.644276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.644247 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd276\" (UniqueName: \"kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276\") pod \"payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.644755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.644720 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.829471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.829437 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:11.879252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.879170 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:11.959996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:11.957805 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796a157b-aa08-4e52-bf87-5d6325dd4af1" path="/var/lib/kubelet/pods/796a157b-aa08-4e52-bf87-5d6325dd4af1/volumes" Oct 31 23:17:12.161663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.161503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:12.178640 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:17:12.178596 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac86da50_1a95_4bd1_adda_981f33721b07.slice/crio-08ab60ce4e45cb6ad0f8777706537aa686b81b891e3844901232fefdf2799bf9 WatchSource:0}: Error finding container 08ab60ce4e45cb6ad0f8777706537aa686b81b891e3844901232fefdf2799bf9: Status 404 returned error can't find the container with id 08ab60ce4e45cb6ad0f8777706537aa686b81b891e3844901232fefdf2799bf9 Oct 31 23:17:12.187747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.186456 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6"] Oct 31 23:17:12.264755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.258029 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6"] Oct 31 23:17:12.267443 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:17:12.267404 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7b3cf1_7cb6_4997_976d_8f8c3d0773bc.slice/crio-2660259cf70a704b801e297c0a8760deca3fb1f72528cab3932568b5bba83c5a WatchSource:0}: Error finding container 2660259cf70a704b801e297c0a8760deca3fb1f72528cab3932568b5bba83c5a: Status 404 returned error can't find the container with id 2660259cf70a704b801e297c0a8760deca3fb1f72528cab3932568b5bba83c5a Oct 31 23:17:12.724868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.724816 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" event={"ID":"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc","Type":"ContainerStarted","Data":"91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a"} Oct 31 23:17:12.725338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.724919 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" event={"ID":"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc","Type":"ContainerStarted","Data":"2660259cf70a704b801e297c0a8760deca3fb1f72528cab3932568b5bba83c5a"} Oct 31 23:17:12.725338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.724954 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:12.727225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.727194 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" event={"ID":"ac86da50-1a95-4bd1-adda-981f33721b07","Type":"ContainerStarted","Data":"599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f"} Oct 31 23:17:12.727346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.727249 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" event={"ID":"ac86da50-1a95-4bd1-adda-981f33721b07","Type":"ContainerStarted","Data":"08ab60ce4e45cb6ad0f8777706537aa686b81b891e3844901232fefdf2799bf9"} Oct 31 23:17:12.727346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.727277 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:12.752881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.752817 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" podStartSLOduration=1.752800352 podStartE2EDuration="1.752800352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:12.743827982 +0000 UTC m=+1658.973382392" watchObservedRunningTime="2026-10-31 23:17:12.752800352 +0000 UTC m=+1658.982354762" Oct 31 23:17:12.765947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:12.765894 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" podStartSLOduration=1.765878104 podStartE2EDuration="1.765878104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:12.762531753 +0000 UTC m=+1658.992086161" watchObservedRunningTime="2026-10-31 23:17:12.765878104 +0000 UTC m=+1658.995432514" Oct 31 23:17:13.109410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:13.109370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:13.857307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:13.857250 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:14.670356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:14.670313 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:15.610928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:15.608111 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:16.358403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:16.358372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:16.981472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:16.981437 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:17.123474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:17.123438 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:17.943082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:17.943043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:18.553998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:18.553961 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:18.737265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:18.737219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:19.338036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:19.337990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:19.917354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:19.917316 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:20.720564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:20.720513 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:20.882063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:20.882020 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:21.049127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:21.049079 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:21.201157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:21.201124 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:21.731933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:21.731898 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:22.027919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:22.027890 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w"] Oct 31 23:17:22.033416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:22.033379 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b3cc357-wfr7w"] Oct 31 23:17:22.545580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:22.542214 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:23.125595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:23.125515 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:23.744539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:23.744511 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:17:23.745025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:23.744597 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:17:23.773329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:23.773295 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:23.963072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:23.963034 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b2e27f-2e7b-4e71-be22-b8cf3c7df578" path="/var/lib/kubelet/pods/f8b2e27f-2e7b-4e71-be22-b8cf3c7df578/volumes" Oct 31 23:17:24.394029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:24.393996 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:24.891390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:24.891341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:25.127314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:25.127265 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:25.995604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:25.995518 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:26.370306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.370219 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l"] Oct 31 23:17:26.375261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.375225 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.385106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.385079 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l"] Oct 31 23:17:26.385871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.385839 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:17:26.387486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.387456 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:17:26.387647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.387631 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:17:26.387733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.387707 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:17:26.388518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.388488 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.388746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.388723 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.388897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.388882 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlv4j\" (UniqueName: \"kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.489986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.489953 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.490165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.490038 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.490165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.490144 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlv4j\" (UniqueName: \"kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.490787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.490765 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.492294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.492273 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.502456 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.502433 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlv4j\" (UniqueName: \"kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j\") pod \"maas-api-revoke-sub-b407cae8-6zt6l\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:26.623397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.623317 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:26.709254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:26.709185 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:27.034620 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:17:27.034537 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3113e8e9_2590_406b_a181_5b52bad97a12.slice/crio-681fc46ec740be2c3bcef4fff712c10bff225758d51148ff9e4be2b772f1708c WatchSource:0}: Error finding container 681fc46ec740be2c3bcef4fff712c10bff225758d51148ff9e4be2b772f1708c: Status 404 returned error can't find the container with id 681fc46ec740be2c3bcef4fff712c10bff225758d51148ff9e4be2b772f1708c Oct 31 23:17:27.036906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:27.036783 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l"] Oct 31 23:17:27.298657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:27.296327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:27.858082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:27.858049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:27.981426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:27.981386 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" event={"ID":"3113e8e9-2590-406b-a181-5b52bad97a12","Type":"ContainerStarted","Data":"25c797e23000023de4bba8307a9d50c245ae8ff0af3dc93404df320e81071ed7"} Oct 31 23:17:27.981694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:27.981458 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" event={"ID":"3113e8e9-2590-406b-a181-5b52bad97a12","Type":"ContainerStarted","Data":"681fc46ec740be2c3bcef4fff712c10bff225758d51148ff9e4be2b772f1708c"} Oct 31 23:17:27.997108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:27.996824 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:28.005226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:28.005174 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" podStartSLOduration=2.005160073 podStartE2EDuration="2.005160073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:27.997375822 +0000 UTC m=+1674.226930231" watchObservedRunningTime="2026-10-31 23:17:28.005160073 +0000 UTC m=+1674.234714482" Oct 31 23:17:28.178498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:28.178469 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:28.698242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:28.698206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:28.990877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:28.990790 5043 generic.go:356] "Generic (PLEG): container finished" podID="3113e8e9-2590-406b-a181-5b52bad97a12" containerID="25c797e23000023de4bba8307a9d50c245ae8ff0af3dc93404df320e81071ed7" exitCode=0 Oct 31 23:17:28.991040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:28.990879 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" event={"ID":"3113e8e9-2590-406b-a181-5b52bad97a12","Type":"ContainerDied","Data":"25c797e23000023de4bba8307a9d50c245ae8ff0af3dc93404df320e81071ed7"} Oct 31 23:17:29.532568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:29.532356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:30.198454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.198418 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:30.288903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.288840 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:30.399739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.399703 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca\") pod \"3113e8e9-2590-406b-a181-5b52bad97a12\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " Oct 31 23:17:30.399917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.399755 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j\" (UniqueName: \"kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j\") pod \"3113e8e9-2590-406b-a181-5b52bad97a12\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " Oct 31 23:17:30.399973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.399930 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca\") pod \"3113e8e9-2590-406b-a181-5b52bad97a12\" (UID: \"3113e8e9-2590-406b-a181-5b52bad97a12\") " Oct 31 23:17:30.400264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.400239 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca" pod "3113e8e9-2590-406b-a181-5b52bad97a12" (UID: "3113e8e9-2590-406b-a181-5b52bad97a12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:17:30.400975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.400952 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:17:30.401930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.401908 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca" pod "3113e8e9-2590-406b-a181-5b52bad97a12" (UID: "3113e8e9-2590-406b-a181-5b52bad97a12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:17:30.402024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.401982 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j" pod "3113e8e9-2590-406b-a181-5b52bad97a12" (UID: "3113e8e9-2590-406b-a181-5b52bad97a12"). InnerVolumeSpecName "kube-api-access-tlv4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:17:30.502385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.502351 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3113e8e9-2590-406b-a181-5b52bad97a12-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:17:30.502385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.502383 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlv4j\" (UniqueName: \"kubernetes.io/projected/3113e8e9-2590-406b-a181-5b52bad97a12-kube-api-access-tlv4j\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:17:30.959471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:30.958566 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:31.018771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:31.018736 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" Oct 31 23:17:31.018964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:31.018744 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l" event={"ID":"3113e8e9-2590-406b-a181-5b52bad97a12","Type":"ContainerDied","Data":"681fc46ec740be2c3bcef4fff712c10bff225758d51148ff9e4be2b772f1708c"} Oct 31 23:17:31.018964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:31.018866 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="681fc46ec740be2c3bcef4fff712c10bff225758d51148ff9e4be2b772f1708c" Oct 31 23:17:31.823568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:31.820498 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:32.494777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:32.494735 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:33.210899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:33.210854 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:33.840453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:33.831827 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:34.027398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:34.027370 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl"] Oct 31 23:17:34.035368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:34.035339 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7a3f8e-cxprl"] Oct 31 23:17:34.537166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:34.537044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:35.045135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:35.045069 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:35.389272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:35.389189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:35.532244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:35.532209 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:35.926081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:35.926045 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:35.947118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:35.947083 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58ee5e4-a240-490e-96ee-5019a606c74e" path="/var/lib/kubelet/pods/c58ee5e4-a240-490e-96ee-5019a606c74e/volumes" Oct 31 23:17:36.092190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:36.092154 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:36.247626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:36.246206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:36.663962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:36.663886 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:36.805691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:36.805655 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:37.146816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.146785 5043 scope.go:122] "RemoveContainer" containerID="5106288b7536c4f1f280665ba2bb4499854d26e0e7155157531d64bdb16b75a5" Oct 31 23:17:37.162583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.162536 5043 scope.go:122] "RemoveContainer" containerID="0dbc2776b0634116d6327fa6d89b011694955e469aa5bfb720307febd416f97f" Oct 31 23:17:37.174441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.174413 5043 scope.go:122] "RemoveContainer" containerID="950e4e3f1b5bc69764dbb29b01483fb7d5a93dc309c844b9260dd23cdcd1ce7a" Oct 31 23:17:37.189977 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.189953 5043 scope.go:122] "RemoveContainer" containerID="64ef7970708d5524eadbaca989e3b2936a69d1b1ca2acea876a74af987040206" Oct 31 23:17:37.200913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.200889 5043 scope.go:122] "RemoveContainer" containerID="82b9edbf43c7962c313cd1f8f5ff4e0082d2323ee296a317fa24fd9ae7d77dda" Oct 31 23:17:37.212404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.212375 5043 scope.go:122] "RemoveContainer" containerID="5b2345c36527582316deada4b89a648b3f0c70d42d474f66d1e3783d87e4a488" Oct 31 23:17:37.222878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.222859 5043 scope.go:122] "RemoveContainer" containerID="9f49382b3d03a17a0f23e56ed1e147acf3676128b5770dc71921af4614c16e31" Oct 31 23:17:37.234926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.234902 5043 scope.go:122] "RemoveContainer" containerID="b410ae72c0e082bd54b5919cfcac129054251986343114860294b6e2274565ff" Oct 31 23:17:37.248023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.247991 5043 scope.go:122] "RemoveContainer" containerID="4246a31c9e64a1776d2e9c2d24fdac728521f241b1f7a291fef282e2de16024c" Oct 31 23:17:37.258510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.258488 5043 scope.go:122] "RemoveContainer" containerID="c700e343e13353fb9d37df3bc3d573ed694a5ac4180842f99bb2c68c463256dc" Oct 31 23:17:37.269843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.269821 5043 scope.go:122] "RemoveContainer" containerID="9170ca7e774508a7ecde8b5f7f42147b60fbd120dda647844c4e7f9d03d8e87a" Oct 31 23:17:37.279445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.279421 5043 scope.go:122] "RemoveContainer" containerID="cab6cbe3af34a613660a08f3c4dfc2c994ea3dfdec42c54535622db8b58e4cfc" Oct 31 23:17:37.293338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.293296 5043 scope.go:122] "RemoveContainer" containerID="7611d804104cff17f9976f7c57186e5997df0521d0b89e04b15ace0326cbf0c0" Oct 31 23:17:37.303873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.303857 5043 scope.go:122] "RemoveContainer" containerID="9531081ba2dde1c7f91be9432dace3a856b98affa0ffa0b2f24033cd16c9f644" Oct 31 23:17:37.313790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.313765 5043 scope.go:122] "RemoveContainer" containerID="7faac352079ca1db03a2e6e17ecd52ae735908e138875f75a2c4d0318e3b35e4" Oct 31 23:17:37.326319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.326284 5043 scope.go:122] "RemoveContainer" containerID="62a66558c63d7fac5b8012c0b89c3040426f91d0da7d776839378ae1d2e270df" Oct 31 23:17:37.338263 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.338244 5043 scope.go:122] "RemoveContainer" containerID="49e62936a733cf2bd5b7b7c25fee9f0a86f7dbdf294432948e8b06f3c6a69566" Oct 31 23:17:37.350672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.350648 5043 scope.go:122] "RemoveContainer" containerID="31f94bcd9ebb0366ebccaadeefdc5d00210beb9d4737a72cb5c5de9b38e5e5a8" Oct 31 23:17:37.363116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.363087 5043 scope.go:122] "RemoveContainer" containerID="6c841bf719b9c5b6096d6bd36f2edf4577f405bb9989a5cfca19089054a5be7b" Oct 31 23:17:37.526519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:37.526484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:38.241568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:38.238506 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:38.976432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:38.973026 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:39.638325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:39.638291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:39.905408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:39.905329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:40.050458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:40.050427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:40.494126 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:17:40.494080 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:17:40.565619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:40.565588 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:41.082297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.082266 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:41.082773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.082597 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds"] Oct 31 23:17:41.083411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.083395 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3113e8e9-2590-406b-a181-5b52bad97a12" containerName="revoke-keys" Oct 31 23:17:41.083449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.083413 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3113e8e9-2590-406b-a181-5b52bad97a12" containerName="revoke-keys" Oct 31 23:17:41.083629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.083617 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3113e8e9-2590-406b-a181-5b52bad97a12" containerName="revoke-keys" Oct 31 23:17:41.091896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.091873 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.104943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.104918 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp"] Oct 31 23:17:41.110137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.108353 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d9e90c\"" Oct 31 23:17:41.110741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.110717 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.112854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.112741 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds"] Oct 31 23:17:41.127757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.127733 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d9e90c-dockercfg-8p8jf\"" Oct 31 23:17:41.129008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.128932 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp"] Oct 31 23:17:41.162793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.162759 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8zx\" (UniqueName: \"kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.163027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.162905 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lh6\" (UniqueName: \"kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6\") pod \"payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.163027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.162940 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.163208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.163075 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.308572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.298927 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:41.317569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.313829 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lh6\" (UniqueName: \"kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6\") pod \"payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.317569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.313883 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.317569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.314024 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.317569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.314263 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8zx\" (UniqueName: \"kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.317569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.315558 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.317569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.316677 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.341617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.340373 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8zx\" (UniqueName: \"kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.393577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.384343 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lh6\" (UniqueName: \"kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6\") pod \"payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.444407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.443133 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:41.459597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.458937 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:41.721032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.720918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:41.937494 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:17:41.937459 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4c1dc7_8374_4d2e_bedf_3a7a39abdfd1.slice/crio-009124b0e480b77a4101338c7eeb6488e0ed9e2adcd6f1d9ff1deb591bcaf950 WatchSource:0}: Error finding container 009124b0e480b77a4101338c7eeb6488e0ed9e2adcd6f1d9ff1deb591bcaf950: Status 404 returned error can't find the container with id 009124b0e480b77a4101338c7eeb6488e0ed9e2adcd6f1d9ff1deb591bcaf950 Oct 31 23:17:41.968835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.965359 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp"] Oct 31 23:17:41.968835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:41.965435 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds"] Oct 31 23:17:42.483210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.483168 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" event={"ID":"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1","Type":"ContainerStarted","Data":"b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01"} Oct 31 23:17:42.483715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.483239 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:42.483715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.483268 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" event={"ID":"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1","Type":"ContainerStarted","Data":"009124b0e480b77a4101338c7eeb6488e0ed9e2adcd6f1d9ff1deb591bcaf950"} Oct 31 23:17:42.485784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.485202 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" event={"ID":"3bf431e0-53a2-4422-905b-ae83975f1052","Type":"ContainerStarted","Data":"acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d"} Oct 31 23:17:42.485784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.485260 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" event={"ID":"3bf431e0-53a2-4422-905b-ae83975f1052","Type":"ContainerStarted","Data":"bcfd963bd6da4582bb533534f1bae554e8476d6ba8858c2e6161ef8560b37d2b"} Oct 31 23:17:42.485784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.485362 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:42.495151 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.495123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:42.581571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.563350 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" podStartSLOduration=1.56333022 podStartE2EDuration="1.56333022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:42.510231578 +0000 UTC m=+1688.739785989" watchObservedRunningTime="2026-10-31 23:17:42.56333022 +0000 UTC m=+1688.792884630" Oct 31 23:17:42.605878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:42.605813 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" podStartSLOduration=1.605793959 podStartE2EDuration="1.605793959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:17:42.542005331 +0000 UTC m=+1688.771559742" watchObservedRunningTime="2026-10-31 23:17:42.605793959 +0000 UTC m=+1688.835348371" Oct 31 23:17:43.215445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:43.215406 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:43.353231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:43.353189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:44.220822 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:44.220788 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:45.007278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:45.007244 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:45.815103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:45.815060 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:46.407001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:46.406966 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:47.033575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:47.021966 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:47.728489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:47.727909 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:48.621031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:48.620996 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:49.289717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:49.289686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:49.456022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:49.455988 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:49.901250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:49.901222 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:50.052335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:50.052276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:50.188416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:50.188328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:50.749209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:50.749119 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:51.318795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:51.318761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:51.534852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:51.534813 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:52.225903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:52.225860 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:52.827183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:52.827099 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:53.501100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:53.495391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:53.540211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:53.540174 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:17:53.541652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:53.541450 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:17:54.201975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:54.201946 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:54.828563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:54.828336 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:55.539196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:55.538128 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:55.728811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:55.728767 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:17:55.729022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:55.728849 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:17:55.995982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:55.995952 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:56.042304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:56.042267 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5"] Oct 31 23:17:56.053940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:56.053855 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d015b7c9-chmr5"] Oct 31 23:17:56.144717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:56.144374 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:56.734909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:56.734876 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:57.496107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.496077 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:57.724981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.724931 5043 generic.go:356] "Generic (PLEG): container finished" podID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerID="5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211" exitCode=0 Oct 31 23:17:57.725200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.725083 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" event={"ID":"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef","Type":"ContainerDied","Data":"5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211"} Oct 31 23:17:57.725933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.725906 5043 scope.go:122] "RemoveContainer" containerID="5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211" Oct 31 23:17:57.729829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.729780 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:17:57.729956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.729933 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:17:57.730057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.730033 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:17:57.763790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.763761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:57.940256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:57.940222 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884c397b-4a5e-4f15-8eef-f77a866bc2ae" path="/var/lib/kubelet/pods/884c397b-4a5e-4f15-8eef-f77a866bc2ae/volumes" Oct 31 23:17:58.221632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:58.215065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:58.761699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:58.760237 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" event={"ID":"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef","Type":"ContainerStarted","Data":"a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c"} Oct 31 23:17:58.816566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:58.810750 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:59.533581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:59.524168 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:17:59.663040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:59.663003 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:17:59.663246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:59.663186 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 23:17:59.663301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:17:59.663248 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 23:18:00.241566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:00.233486 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:00.663788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:00.663697 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 23:18:00.663788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:00.663754 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 23:18:01.078217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:01.078187 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:01.662897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:01.662866 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 23:18:01.663324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:01.662913 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 23:18:01.904605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:01.904568 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:02.523061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:02.523020 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:02.663542 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:02.663504 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 23:18:02.663990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:02.663608 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 23:18:03.247885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.247846 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:03.394215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.394183 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:03.663609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.663503 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 23:18:03.663609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.663590 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 23:18:03.768156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.768129 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr"] Oct 31 23:18:03.810355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.810322 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr"] Oct 31 23:18:03.810639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.810613 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:03.815249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.815213 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-2126ff-openshift-default-dockercfg-6g865\"" Oct 31 23:18:03.840161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:03.840101 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:04.033570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.032997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:04.033776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.033746 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.033848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.033794 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.033906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.033888 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.033969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.033914 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6jk\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.034054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.034041 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.034109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.034081 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.045580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.034298 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.045580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.034387 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.045580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.034481 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.101573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.098798 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw"] Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139454 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139575 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139645 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139677 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139773 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139801 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6jk\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139933 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.139974 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.144567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.140079 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.149416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.149057 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.169215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.149862 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.169215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.150114 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.169215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.150797 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.186772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.186739 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.188832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.188772 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh"] Oct 31 23:18:04.230578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.204065 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.230578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.207049 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.274610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.260238 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.280445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.279093 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6jk\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk\") pod \"e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.323757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.323670 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-affc41c5-d5qbw"] Oct 31 23:18:04.334661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.334630 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d559f2a-cb5fh"] Oct 31 23:18:04.435566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.430866 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:04.668038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.667957 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:18:04.668472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.668322 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:18:04.669030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.669012 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:18:04.742831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:04.742580 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:05.181120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.181088 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:05.206251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.206141 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr"] Oct 31 23:18:05.221516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.221471 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:18:05.224056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.224026 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:18:05.224242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.224225 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:18:05.400596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.399826 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:05.899058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.899021 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" event={"ID":"74083906-8df2-4339-9148-e0d1f5c4de69","Type":"ContainerStarted","Data":"43d680ee9d1ab3f662447e423976d1a941a1f04d2a364349d62ca1799ea498a5"} Oct 31 23:18:05.899592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.899082 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" event={"ID":"74083906-8df2-4339-9148-e0d1f5c4de69","Type":"ContainerStarted","Data":"ae73db52c17d95f898b8eaf91c9a5714a27791b9a25c04c1fd5bfbf9260d09f3"} Oct 31 23:18:05.926076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.926023 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podStartSLOduration=2.926009036 podStartE2EDuration="2.926009036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:05.923434437 +0000 UTC m=+1712.152988850" watchObservedRunningTime="2026-10-31 23:18:05.926009036 +0000 UTC m=+1712.155563445" Oct 31 23:18:05.942066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.942036 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fea76e4-fb99-4653-9529-1f2c8357e7a9" path="/var/lib/kubelet/pods/8fea76e4-fb99-4653-9529-1f2c8357e7a9/volumes" Oct 31 23:18:05.942595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:05.942537 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f16cb3a-edc2-461e-8367-31df7508a25a" path="/var/lib/kubelet/pods/9f16cb3a-edc2-461e-8367-31df7508a25a/volumes" Oct 31 23:18:06.127596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.119499 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:06.435639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.435600 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:06.442810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.442783 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:06.554266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.554228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:06.794051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.794013 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:06.908985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.908954 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:06.909848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.909823 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:18:06.936055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:06.936019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:07.801259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:07.801229 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:07.949243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:07.949198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:08.126249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:08.108245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:08.326196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:08.326166 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:08.470286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:08.470206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:08.827578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:08.825093 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:09.387259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.387225 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n"] Oct 31 23:18:09.395644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.395613 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.400208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.400184 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:18:09.400654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.400627 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n"] Oct 31 23:18:09.401638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.401616 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:18:09.401758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.401640 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:18:09.402844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.402824 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:18:09.456535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.456501 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.457157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.457127 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7txg\" (UniqueName: \"kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.459136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.459111 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.464987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.464960 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:09.526530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.526493 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk"] Oct 31 23:18:09.532939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.532915 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.542636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.542613 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2126ff\"" Oct 31 23:18:09.560692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.560660 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb"] Oct 31 23:18:09.566009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.565970 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.566182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.566032 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7txg\" (UniqueName: \"kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.566182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.566086 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5bq\" (UniqueName: \"kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq\") pod \"payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.566310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.566265 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.566419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.566398 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.567326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.567299 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.568933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.568913 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk"] Oct 31 23:18:09.569050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.569024 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.581607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.575660 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.581607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.577207 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2126ff-dockercfg-xf2s4\"" Oct 31 23:18:09.581607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.577578 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb"] Oct 31 23:18:09.583175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.583151 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7txg\" (UniqueName: \"kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg\") pod \"maas-api-revoke-sub-285962ef-5bd6n\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.605073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.605040 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:09.688625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.675081 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.688625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.675279 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.688625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.675347 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5bq\" (UniqueName: \"kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq\") pod \"payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.688625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.675386 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jz6j\" (UniqueName: \"kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j\") pod \"payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.706571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.700671 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.758656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.734511 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:09.766568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.761219 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5bq\" (UniqueName: \"kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq\") pod \"payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.798568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.794534 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.824581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.822627 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.824581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.822884 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jz6j\" (UniqueName: \"kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j\") pod \"payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.878646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.872877 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:09.899569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.886602 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jz6j\" (UniqueName: \"kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j\") pod \"payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:09.970307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:09.967538 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:10.063566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:10.063377 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh"] Oct 31 23:18:10.093644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:10.093310 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42ffee0c-sldkh"] Oct 31 23:18:10.364567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:10.348903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:10.471685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:10.470594 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n"] Oct 31 23:18:10.477571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:10.477531 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb"] Oct 31 23:18:10.515251 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:18:10.515220 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:18:10.625245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:10.625222 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk"] Oct 31 23:18:10.717065 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:18:10.717003 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1072fa_4c18_4b72_a947_325fa267a72f.slice/crio-c3060fe7f2f40501d646e80f948f10e277932b6f23ce8f5de857c33c1da6fe8b WatchSource:0}: Error finding container c3060fe7f2f40501d646e80f948f10e277932b6f23ce8f5de857c33c1da6fe8b: Status 404 returned error can't find the container with id c3060fe7f2f40501d646e80f948f10e277932b6f23ce8f5de857c33c1da6fe8b Oct 31 23:18:11.008918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.008083 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:11.051442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.050171 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq"] Oct 31 23:18:11.058835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.058798 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4536121-qq6qq"] Oct 31 23:18:11.128788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.128665 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" event={"ID":"3db4e830-7446-4841-9080-f4572baaecca","Type":"ContainerStarted","Data":"90e64e438bd45a3b37d8dc8a0f2035c3c8c2b25943e3e7281e357f39ed4e628c"} Oct 31 23:18:11.128788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.128734 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:11.128788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.128756 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" event={"ID":"3db4e830-7446-4841-9080-f4572baaecca","Type":"ContainerStarted","Data":"5507131c7ff3f28eb790d772235502741f6da87f7eee3f73f2c790cfda96a6f9"} Oct 31 23:18:11.131362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.131330 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" event={"ID":"ce21a44f-8c15-4e6e-9cfd-f01479405787","Type":"ContainerStarted","Data":"aafb50876f9cc8f75c1fdadf9b683c05e3b552873b6b0e43816a7b4e5d3e4117"} Oct 31 23:18:11.131535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.131385 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" event={"ID":"ce21a44f-8c15-4e6e-9cfd-f01479405787","Type":"ContainerStarted","Data":"02ddbdd378e9506af0bf61922ca7a10fc9b0c4618717a729b3e75c640baaaa03"} Oct 31 23:18:11.140425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.140393 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" event={"ID":"7e1072fa-4c18-4b72-a947-325fa267a72f","Type":"ContainerStarted","Data":"ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f"} Oct 31 23:18:11.140631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.140450 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" event={"ID":"7e1072fa-4c18-4b72-a947-325fa267a72f","Type":"ContainerStarted","Data":"c3060fe7f2f40501d646e80f948f10e277932b6f23ce8f5de857c33c1da6fe8b"} Oct 31 23:18:11.140631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.140479 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:11.164161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.164104 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" podStartSLOduration=2.16408681 podStartE2EDuration="2.16408681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:11.154071413 +0000 UTC m=+1717.383625823" watchObservedRunningTime="2026-10-31 23:18:11.16408681 +0000 UTC m=+1717.393641220" Oct 31 23:18:11.176126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.176071 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podStartSLOduration=2.176057081 podStartE2EDuration="2.176057081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:11.174466919 +0000 UTC m=+1717.404021331" watchObservedRunningTime="2026-10-31 23:18:11.176057081 +0000 UTC m=+1717.405611492" Oct 31 23:18:11.199326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.199280 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" podStartSLOduration=2.199267938 podStartE2EDuration="2.199267938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:11.195675328 +0000 UTC m=+1717.425229736" watchObservedRunningTime="2026-10-31 23:18:11.199267938 +0000 UTC m=+1717.428822375" Oct 31 23:18:11.691397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.691367 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:11.831275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.831242 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:11.943446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.943376 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea6b8e4-d3b1-48f7-b502-a97009730d7f" path="/var/lib/kubelet/pods/7ea6b8e4-d3b1-48f7-b502-a97009730d7f/volumes" Oct 31 23:18:11.944197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:11.944176 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac2704b-bdca-4833-9fd4-dc78917c9c03" path="/var/lib/kubelet/pods/9ac2704b-bdca-4833-9fd4-dc78917c9c03/volumes" Oct 31 23:18:12.074537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.074503 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml"] Oct 31 23:18:12.081181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.081159 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.100309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.100283 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml"] Oct 31 23:18:12.123300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.123268 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-85b90bac-openshift-default-dockercfg-lqfl8\"" Oct 31 23:18:12.160945 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.160519 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.162237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.162191 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.163187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.162990 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.164014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.163818 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.164408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.164389 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.164732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.164711 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.165084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.165066 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.165330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.165311 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.165558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.165527 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.311582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.308615 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330340 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330415 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330448 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330480 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330586 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330732 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.330939 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.331005 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.331065 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.331561 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.331810 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.334575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.332022 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.370679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.351210 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.402591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.402497 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.420020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.419380 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.443242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.428830 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.526799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.526750 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:12.531994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.531008 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.550526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.550379 5043 generic.go:356] "Generic (PLEG): container finished" podID="ce21a44f-8c15-4e6e-9cfd-f01479405787" containerID="aafb50876f9cc8f75c1fdadf9b683c05e3b552873b6b0e43816a7b4e5d3e4117" exitCode=0 Oct 31 23:18:12.552035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.552007 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" event={"ID":"ce21a44f-8c15-4e6e-9cfd-f01479405787","Type":"ContainerDied","Data":"aafb50876f9cc8f75c1fdadf9b683c05e3b552873b6b0e43816a7b4e5d3e4117"} Oct 31 23:18:12.578204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.578126 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo\") pod \"e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.705901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.705871 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:12.724666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:12.724501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:13.081849 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:18:13.081811 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a2e7de_778e_47c1_a35a_d4f066df6748.slice/crio-0467786627e3156d624c9a0bc2db4a47fe5787e3e810e9d043ca61837d0aca54 WatchSource:0}: Error finding container 0467786627e3156d624c9a0bc2db4a47fe5787e3e810e9d043ca61837d0aca54: Status 404 returned error can't find the container with id 0467786627e3156d624c9a0bc2db4a47fe5787e3e810e9d043ca61837d0aca54 Oct 31 23:18:13.082697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.082664 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml"] Oct 31 23:18:13.085250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.085214 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:18:13.085335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.085294 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:18:13.085374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.085335 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:18:13.368066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.367957 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:13.585346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.585296 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" event={"ID":"b6a2e7de-778e-47c1-a35a-d4f066df6748","Type":"ContainerStarted","Data":"a112283e701bccf3ade34448bb910c025ecbd7961915ee71787f35773df4709c"} Oct 31 23:18:13.585446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.585358 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" event={"ID":"b6a2e7de-778e-47c1-a35a-d4f066df6748","Type":"ContainerStarted","Data":"0467786627e3156d624c9a0bc2db4a47fe5787e3e810e9d043ca61837d0aca54"} Oct 31 23:18:13.613681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.611751 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" podStartSLOduration=1.611734118 podStartE2EDuration="1.611734118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:13.609569865 +0000 UTC m=+1719.839124278" watchObservedRunningTime="2026-10-31 23:18:13.611734118 +0000 UTC m=+1719.841288529" Oct 31 23:18:13.706977 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.706946 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:13.876454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.874965 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:13.907426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.907389 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca\") pod \"ce21a44f-8c15-4e6e-9cfd-f01479405787\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " Oct 31 23:18:13.907617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.907536 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca\") pod \"ce21a44f-8c15-4e6e-9cfd-f01479405787\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " Oct 31 23:18:13.907698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.907627 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg\" (UniqueName: \"kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg\") pod \"ce21a44f-8c15-4e6e-9cfd-f01479405787\" (UID: \"ce21a44f-8c15-4e6e-9cfd-f01479405787\") " Oct 31 23:18:13.907929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.907882 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca" pod "ce21a44f-8c15-4e6e-9cfd-f01479405787" (UID: "ce21a44f-8c15-4e6e-9cfd-f01479405787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:18:13.908598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.908575 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:18:13.909864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.909813 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca" pod "ce21a44f-8c15-4e6e-9cfd-f01479405787" (UID: "ce21a44f-8c15-4e6e-9cfd-f01479405787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:18:13.910121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:13.910107 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg" pod "ce21a44f-8c15-4e6e-9cfd-f01479405787" (UID: "ce21a44f-8c15-4e6e-9cfd-f01479405787"). InnerVolumeSpecName "kube-api-access-z7txg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:18:14.010554 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.010520 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7txg\" (UniqueName: \"kubernetes.io/projected/ce21a44f-8c15-4e6e-9cfd-f01479405787-kube-api-access-z7txg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:18:14.010704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.010566 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce21a44f-8c15-4e6e-9cfd-f01479405787-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:18:14.124124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.124080 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:14.595224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.595195 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" Oct 31 23:18:14.595224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.595220 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n" event={"ID":"ce21a44f-8c15-4e6e-9cfd-f01479405787","Type":"ContainerDied","Data":"02ddbdd378e9506af0bf61922ca7a10fc9b0c4618717a729b3e75c640baaaa03"} Oct 31 23:18:14.595451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.595284 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ddbdd378e9506af0bf61922ca7a10fc9b0c4618717a729b3e75c640baaaa03" Oct 31 23:18:14.713781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.713749 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:14.800170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:14.799349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:15.596902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:15.596857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:15.653466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:15.651927 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:15.653466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:15.653196 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:18:15.877003 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:15.876920 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:16.015877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:16.015842 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:16.370715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:16.370681 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:17.363637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.363599 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:17.416943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.416904 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm"] Oct 31 23:18:17.418244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.418212 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ce21a44f-8c15-4e6e-9cfd-f01479405787" containerName="revoke-keys" Oct 31 23:18:17.418244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.418243 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce21a44f-8c15-4e6e-9cfd-f01479405787" containerName="revoke-keys" Oct 31 23:18:17.418594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.418574 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce21a44f-8c15-4e6e-9cfd-f01479405787" containerName="revoke-keys" Oct 31 23:18:17.425497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.424341 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:17.450634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.446659 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm"] Oct 31 23:18:17.477485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.477170 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-85b90bac\"" Oct 31 23:18:17.477485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.477351 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-85b90bac-dockercfg-mfhg6\"" Oct 31 23:18:17.540023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.539982 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28hd\" (UniqueName: \"kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd\") pod \"payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:17.540207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.540094 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume\") pod \"payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:17.648627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.646019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:17.884567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.883084 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28hd\" (UniqueName: \"kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd\") pod \"payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:17.884567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.883205 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume\") pod \"payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:17.897777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.897203 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume\") pod \"payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:17.946574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:17.946470 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28hd\" (UniqueName: \"kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd\") pod \"payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:18.216817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:18.214498 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:18.414564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:18.412578 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:18.821027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:18.810761 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm"] Oct 31 23:18:18.968559 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:18:18.956688 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06197912_3b3e_40aa_a19b_8025350bf804.slice/crio-8802eea9293b8abe3d56ca03197700a8026ef5a64403360c0bb4303d9085d089 WatchSource:0}: Error finding container 8802eea9293b8abe3d56ca03197700a8026ef5a64403360c0bb4303d9085d089: Status 404 returned error can't find the container with id 8802eea9293b8abe3d56ca03197700a8026ef5a64403360c0bb4303d9085d089 Oct 31 23:18:19.227575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:19.225735 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" event={"ID":"06197912-3b3e-40aa-a19b-8025350bf804","Type":"ContainerStarted","Data":"8802eea9293b8abe3d56ca03197700a8026ef5a64403360c0bb4303d9085d089"} Oct 31 23:18:19.705410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:19.705370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:19.862176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:19.862037 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:20.019574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.018047 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:20.087576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.083773 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:20.229620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.229355 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:20.409570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.409200 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:18:20.440741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.419634 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:20.516570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.504011 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" event={"ID":"06197912-3b3e-40aa-a19b-8025350bf804","Type":"ContainerStarted","Data":"6bf56397124ce158b9f46a647c9478377b71d2ee07f31193a58e705fc692976a"} Oct 31 23:18:20.516570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.514736 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:20.556570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.545464 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podStartSLOduration=3.545445209 podStartE2EDuration="3.545445209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:20.542578274 +0000 UTC m=+1726.772132897" watchObservedRunningTime="2026-10-31 23:18:20.545445209 +0000 UTC m=+1726.774999620" Oct 31 23:18:20.836621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:20.832689 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:21.044479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.044439 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:21.229621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.219926 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:21.236090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.231264 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:21.443662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.443627 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:21.762571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.761983 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:18:21.768577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.764606 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:18:21.888573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.841063 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:21.888573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.851889 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:18:21.889626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:21.889515 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:22.146629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.142282 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:18:22.176272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.158632 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:22.176272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.164407 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:22.223661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.223615 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:22.229713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.229676 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:22.290595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.290560 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:18:22.341040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.341010 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:18:22.456679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.455697 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:18:22.573939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.567380 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:18:22.575074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.574783 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:18:22.693848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:22.693812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:23.600558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:23.600503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:24.426570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:24.422786 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:24.962902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:24.962867 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:25.133444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:25.133403 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:25.297858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:25.297821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:26.019507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:26.019437 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:26.899989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:26.899950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:27.514784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:27.514745 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:27.653926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:27.653893 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:28.118481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:28.118446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:28.819076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:28.819044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:28.956130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:28.956095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:29.100837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:29.100761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:29.649800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:29.649769 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:30.301298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:30.301254 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:31.017191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:31.017147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:31.634512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:31.634435 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:31.941963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:31.941888 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:18:32.033959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:32.033922 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq"] Oct 31 23:18:32.043215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:32.043186 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cc6360c-pshrq"] Oct 31 23:18:32.330356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:32.330322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:33.023491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.023454 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:33.035619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.035564 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw"] Oct 31 23:18:33.041731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.041703 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f590727-cgmrw"] Oct 31 23:18:33.046957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.046933 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2"] Oct 31 23:18:33.051862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.051841 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3fc202-8fsp2"] Oct 31 23:18:33.605301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.605255 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:33.944896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.944794 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4dfdd2-48b9-4613-aee0-18e7090394d8" path="/var/lib/kubelet/pods/3f4dfdd2-48b9-4613-aee0-18e7090394d8/volumes" Oct 31 23:18:33.945529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.945508 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48eab729-858d-47b4-a23a-a4cb42abfa40" path="/var/lib/kubelet/pods/48eab729-858d-47b4-a23a-a4cb42abfa40/volumes" Oct 31 23:18:33.946147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:33.946128 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be45fe22-12e1-43fe-8192-ec1028054740" path="/var/lib/kubelet/pods/be45fe22-12e1-43fe-8192-ec1028054740/volumes" Oct 31 23:18:34.028640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:34.028604 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v"] Oct 31 23:18:34.034786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:34.034759 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13bbe0d3-hb58v"] Oct 31 23:18:34.522423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:34.522389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:35.203614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:35.203504 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:35.803220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:35.803115 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:35.944673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:35.944633 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aff63cd-52bc-42ec-993f-969f8d7339b5" path="/var/lib/kubelet/pods/7aff63cd-52bc-42ec-993f-969f8d7339b5/volumes" Oct 31 23:18:36.443455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:36.443405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:37.115603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.115565 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:37.525950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.525920 5043 scope.go:122] "RemoveContainer" containerID="d15e5123c124c030b74ac62b1d773e4b45229cd4d1bfa26d3b2368c9ca14f189" Oct 31 23:18:37.540163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.540140 5043 scope.go:122] "RemoveContainer" containerID="f9badd16f34b7a57c1a5697b80f2f3279d3521048a22efa7436090d022cfbafd" Oct 31 23:18:37.554044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.554020 5043 scope.go:122] "RemoveContainer" containerID="3ce0927ddc223e937937853d98509e409aa10f726fa807647825ead8144ca080" Oct 31 23:18:37.566884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.566859 5043 scope.go:122] "RemoveContainer" containerID="a827f873aaa4a2f20c2cbda89cd9e3e9dd3631275f4a11b92a8d60b2ca5c8a66" Oct 31 23:18:37.577951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.577934 5043 scope.go:122] "RemoveContainer" containerID="6050e19026abbfd05d532c1d7a11bb04d493d695f770512ded4fe0de3a0298d0" Oct 31 23:18:37.795503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:37.795416 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:38.524854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:38.517214 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:38.788493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:38.788393 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:39.269312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:39.269277 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:39.406039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:39.406008 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:39.730971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:39.730883 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:39.911264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:39.911158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:40.078572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:40.075623 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx"] Oct 31 23:18:40.127274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:40.119798 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cace0450-qnjfx"] Oct 31 23:18:40.302081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:40.302049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:40.483296 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:18:40.483202 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:18:40.989928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:40.989870 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:41.942935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:41.942903 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3075e7d2-db72-431f-90c9-34fcfea0e9a9" path="/var/lib/kubelet/pods/3075e7d2-db72-431f-90c9-34fcfea0e9a9/volumes" Oct 31 23:18:41.962926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:41.962884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:42.705677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:42.705637 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:43.377303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:43.377261 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:43.559952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:43.559918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:44.354341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.354212 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:44.854744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.854702 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:44.962852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.962818 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx"] Oct 31 23:18:44.968010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.967989 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:44.974252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.973650 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-270e8c\"" Oct 31 23:18:44.974252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.973762 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-270e8c-dockercfg-msbww\"" Oct 31 23:18:44.980121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:44.980097 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx"] Oct 31 23:18:45.143202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.142708 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:45.143202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.142830 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tp7f\" (UniqueName: \"kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f\") pod \"payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:45.193568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.193331 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:45.265570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.264681 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:45.265570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.264795 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tp7f\" (UniqueName: \"kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f\") pod \"payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:45.278956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.275498 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:45.364288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.363623 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq"] Oct 31 23:18:45.418611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.405335 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tp7f\" (UniqueName: \"kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f\") pod \"payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:45.471696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.466286 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk"] Oct 31 23:18:45.478240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.478185 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cff7ca-kwzzq"] Oct 31 23:18:45.490134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.488950 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8c3615-x2wpk"] Oct 31 23:18:45.539323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.539287 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:45.589364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:45.589333 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:46.267910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:46.264998 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df1b859-5913-4192-8be1-5a8577bdc554" path="/var/lib/kubelet/pods/7df1b859-5913-4192-8be1-5a8577bdc554/volumes" Oct 31 23:18:46.267910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:46.265946 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0320a6d-9fd8-4704-b0f7-5db4bfc6307a" path="/var/lib/kubelet/pods/d0320a6d-9fd8-4704-b0f7-5db4bfc6307a/volumes" Oct 31 23:18:46.316598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:46.307892 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx"] Oct 31 23:18:46.316598 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:18:46.311996 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d56bf0b_7968_4952_9917_ba0bc5cba955.slice/crio-82627e76da4027d3a45d66ec6de7e4e8a7547632272d3a88a6530c1ec4a568a4 WatchSource:0}: Error finding container 82627e76da4027d3a45d66ec6de7e4e8a7547632272d3a88a6530c1ec4a568a4: Status 404 returned error can't find the container with id 82627e76da4027d3a45d66ec6de7e4e8a7547632272d3a88a6530c1ec4a568a4 Oct 31 23:18:46.361620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:46.361575 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:46.557563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:46.557053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:46.565303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:46.565216 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" event={"ID":"1d56bf0b-7968-4952-9917-ba0bc5cba955","Type":"ContainerStarted","Data":"82627e76da4027d3a45d66ec6de7e4e8a7547632272d3a88a6530c1ec4a568a4"} Oct 31 23:18:47.245643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.245602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:47.460567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.451666 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:47.636279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.636191 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" event={"ID":"1d56bf0b-7968-4952-9917-ba0bc5cba955","Type":"ContainerStarted","Data":"b55434f93433197d088c4b7dd6b426fab9a15158df78e6eee9ce851067bddf0e"} Oct 31 23:18:47.650870 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.650838 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:18:47.730640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.730574 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podStartSLOduration=3.730531916 podStartE2EDuration="3.730531916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:18:47.705920657 +0000 UTC m=+1753.935475068" watchObservedRunningTime="2026-10-31 23:18:47.730531916 +0000 UTC m=+1753.960086328" Oct 31 23:18:47.764769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.764441 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:47.920532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:47.920426 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:48.876574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:48.875597 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:49.083399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:49.083364 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w"] Oct 31 23:18:49.228630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:49.206782 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ad6e216-xrs5w"] Oct 31 23:18:49.928645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:49.928606 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:50.025572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.010158 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3f856c-94b7-4519-8b01-1b67fb76afb1" path="/var/lib/kubelet/pods/ac3f856c-94b7-4519-8b01-1b67fb76afb1/volumes" Oct 31 23:18:50.025890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.025852 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.058686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.058636 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.151843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.151800 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.193034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.179626 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:18:50.225568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.223631 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:18:50.266569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.262984 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.368020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.364955 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:18:50.405739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.405690 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.419565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.418625 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.434736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.433821 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.477656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.476678 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.561661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.560648 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:50.615566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.599416 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:50.656819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:50.656774 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.013768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.013721 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.071533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.071469 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:18:51.073314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.072125 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:18:51.170618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.154340 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:18:51.170618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.154751 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:18:51.192593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.180932 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.192593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.181302 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:18:51.256148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.256102 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:51.258052 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.258026 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.265709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.265530 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:18:51.422889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.387797 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.422889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.388425 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:18:51.450610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.431657 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:18:51.450610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.433276 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.450610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.433704 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:18:51.450610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.433764 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:18:51.487576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.477790 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:51.487576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.478146 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:18:51.487576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.479955 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:18:51.487576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.480628 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:18:51.487576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.480687 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:18:51.487576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.480721 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:18:51.798450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.798404 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:18:52.272849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:51.959834 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:18:52.398590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.205094 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:52.404449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.206498 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:18:52.404688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.309173 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:52.404878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.382479 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:52.405049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.386135 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:18:52.405253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.404790 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:18:52.449942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.449892 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:52.478570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.474963 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:52.478570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.477957 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:18:52.478570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.478007 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:18:52.478570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.478039 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:18:52.478570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.478069 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:18:52.478570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.478096 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:18:52.484592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.480291 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:52.496466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.494823 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:18:52.507682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.498657 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:18:52.507682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.506993 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:18:52.508200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.507877 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:18:52.510197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.510176 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:18:52.564460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.564383 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:18:52.598900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.598873 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:18:52.600698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.600674 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:18:52.601037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.601023 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:18:52.893071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:52.892988 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:53.824572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:53.822540 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:54.514921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:54.514854 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:55.290598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:55.290542 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:55.717011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:55.716923 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:55.972238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:55.972133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:56.774807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:56.774763 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:57.311406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:57.311358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:57.474572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:57.471773 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:58.117582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:58.117469 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:58.552769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:58.552687 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:58.874130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:58.874007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:59.311712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:59.311670 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:18:59.312123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:59.311758 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:18:59.617472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:59.617398 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:18:59.873755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:18:59.873660 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:00.068911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.036661 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:00.349733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.349680 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:00.917732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.508743 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:19:00.917732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.768285 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:19:00.917732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.822702 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:19:01.618231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.986450 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:19:01.618231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:00.987277 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:01.618231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:01.347669 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:01.618231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:01.348643 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:01.618231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:01.349627 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:02.903479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:01.620589 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:19:02.903479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:02.359035 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:19:04.733196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:04.733138 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:19:04.770787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:04.733817 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:19:04.770787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:04.733892 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:19:04.770787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:04.735321 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:19:04.770787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:04.744826 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.41:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:04.770787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:04.744858 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.666491 5043 trace.go:236] Trace[1975453070]: "Calculate volume metrics of kube-api-access-jdpbl for pod kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" (31-Oct-2026 23:19:23.852) (total time: 2098ms): Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: Trace[1975453070]: [2.098301967s] [2.098301967s] END Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.679058 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.720743 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.721065 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.775131 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.775421 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.775661 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.775841 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.776005 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.776387 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.860703 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.860749 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.863256 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.863296 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.863806 5043 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xbgvl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.863852 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.151:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.864112 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.11:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.864144 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.874794 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.880148 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.356239288s: [/var/lib/containers/storage/overlay/fd4926e43034dfa650d495723e50a8c4f751588d119c241983cc3a46fdc13001/diff /var/log/pods/e2e-models-e2e-worker-w3-67fce9_distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8_985fa972-caf5-46b0-be61-ae1fabb59bfe/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.892001 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.895024 5043 trace.go:236] Trace[1898719438]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" (31-Oct-2026 23:19:25.837) (total time: 1057ms): Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: Trace[1898719438]: [1.057323243s] [1.057323243s] END Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.932282 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.933431 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.933456 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.933512 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.933533 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.966704 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.971658 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.971702 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.971770 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:26.971773 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.054581 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.054656 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.061225 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.063646 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.063755 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.099654 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.099741 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.119664 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.120652 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.333619 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.359689 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.366180 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.366272 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.368344 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.368398 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.372263 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.372316 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.382641 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.382703 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.383759 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.383803 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:19:27.431262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.514347 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.629814 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.293206918s: [/var/lib/containers/storage/overlay/ee72b61010a9a1ea533849e18044181436900329a7671283774633f0c006fbc6/diff /var/log/pods/openshift-ingress_e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8_5f5396c0-d04b-4150-9ad9-179d7319cdb4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.629861 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.253956179s: [/var/lib/containers/storage/overlay/4180d7ca734f6611013b65f61b4754c1087730165b8b770b0d89cb563198db27/diff /var/log/pods/openshift-ingress_e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl_f9bb011d-a74e-4d78-8a04-79f7eb1dff8f/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.630444 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.286507695s: [/var/lib/containers/storage/overlay/362dad0059fc4679f23d0be7e9d1c4f326e7bb7ef5e8fb269f69524eb4052bd4/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr_74083906-8df2-4339-9148-e0d1f5c4de69/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.683409 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.702241 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.703905 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.703962 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.704491 5043 logging.go:55] [core] [Channel #646 SubChannel #647]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.165:50052", ServerName: "10.132.0.165:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:48454->10.132.0.165:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.704924 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.175:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.705402 5043 logging.go:55] [core] [Channel #656 SubChannel #673]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.175:50052", ServerName: "10.132.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.175:50052: operation was canceled" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.709035 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.709102 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.712410 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.713608 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.177:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.714719 5043 logging.go:55] [core] [Channel #645 SubChannel #677]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.177:50052", ServerName: "10.132.0.177:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.177:50052: operation was canceled" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.715982 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.170:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.726594 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.726626 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.727662 5043 patch_prober.go:29] interesting pod/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.727693 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.727767 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.181:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.735009 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.760696 5043 logging.go:55] [core] [Channel #658 SubChannel #659]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.170:50052", ServerName: "10.132.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:33624->10.132.0.170:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.764724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.770570 5043 logging.go:55] [core] [Channel #650 SubChannel #657]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.181:50052", ServerName: "10.132.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:41922->10.132.0.181:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.782290 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.788629 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.788955 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789198 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789247 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789402 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789586 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789644 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789864 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.789905 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.790515 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.178:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.814115 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.816154 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.257970133s: [/var/lib/containers/storage/overlay/0c8ae7af32f635c6c2b989595ac2f0e6489c2608eb7759e58b30bd471893502e/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-x9wxs_11dc44da-2acd-4370-a0c7-37b534779995/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.823527 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.840682 5043 logging.go:55] [core] [Channel #644 SubChannel #690]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: operation was canceled" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.864747 5043 logging.go:55] [core] [Channel #651 SubChannel #653]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:34848->10.132.0.59:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.914853 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.933904 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.933960 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.938497 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.939335 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.939678 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.946944 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.947264 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.169:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.947375 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.947398 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.947654 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.947681 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.948720 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.948751 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.949006 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.949029 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.960290 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.41:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.960328 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.969279 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.988343 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:27.988711 5043 logging.go:55] [core] [Channel #652 SubChannel #654]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.169:50052", ServerName: "10.132.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:60468->10.132.0.169:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.993231 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.993499 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:27.994851 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.27464372s: [/var/lib/containers/storage/overlay/582f92af76b795b913a9d7bbbd79c004e98fbb0e5bdec8368a17f8b7369e92dc/diff /var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4vgvf_677d053b-fbf4-4b76-9782-6b3235dc340f/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.006018 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.11:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.006051 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.527325 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.527774 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": context deadline exceeded" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.527815 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.528690 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.529123 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.529330 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.172:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.529510 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.171:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.529515 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.185:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.538511 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.607574785s: [/var/lib/containers/storage/overlay/5ee1a95b8c2f0a9de349af82eef785d833f2bd6c3628d3cad3e8ab2fe981f6d0/diff /var/log/pods/openshift-ingress_e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx_9a79e0dc-3d9b-4309-af30-92c2d76e8c41/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.538879 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.663423644s: [/var/lib/containers/storage/overlay/b66f341703a742f5e2f5790d520d61f753598b2dc5de4f442a96a6e82ab4b845/diff /var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7_0f223a32-eac0-491e-ae99-0330a8043011/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.544253 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.544304 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.544627 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.6136957s: [/var/lib/containers/storage/overlay/d37f16b86be5aa219373c6dcf0c36b7202c8461f3706428cc39ff2ef63ee91ca/diff /var/log/pods/openshift-ingress_e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb_33eab6b9-e331-48da-bfbf-0d04e0baf215/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.546291 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.546340 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.548568 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.548619 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.550652 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.674770037s: [/var/lib/containers/storage/overlay/91424df212134696fdb2957a411b2c6f07b00f8049686fead10f8a27c01897d6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4_f852be0e-4036-4565-a1c1-d20c99495370/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.551919 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.675578845s: [/var/lib/containers/storage/overlay/d7864d9626b5a9aa77720c0054bae8c20d9fc9d466257229fcb158e4dd6e7a44/diff /var/log/pods/openshift-ingress_payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm_06197912-3b3e-40aa-a19b-8025350bf804/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.552033 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.552074 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.553433 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.553464 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.556670 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.623504925s: [/var/lib/containers/storage/overlay/f9f2e244e27deee91fbf4ef5c76a27749570ca0c594308a2dcb8f938aa66d14a/diff /var/log/pods/openshift-ingress_e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p_084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:28.570890 5043 logging.go:55] [core] [Channel #660 SubChannel #661]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.172:50052", ServerName: "10.132.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:43228->10.132.0.172:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.582415 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.582480 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.589695 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.590204 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:28.590898 5043 logging.go:55] [core] [Channel #664 SubChannel #665]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.185:50052", ServerName: "10.132.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:48490->10.132.0.185:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:28.597057 5043 logging.go:55] [core] [Channel #666 SubChannel #668]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.171:50052", ServerName: "10.132.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:48146->10.132.0.171:50052: use of closed network connection" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.599049 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.665698135s: [/var/lib/containers/storage/overlay/620761409a5e61b66d214418edfcf6a0d03e16658e918faa0146ae0f6025fb4d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb_3db4e830-7446-4841-9080-f4572baaecca/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.599099 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.599144 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.742674 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.745660 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.745724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.765247 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.765294 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.766573 5043 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.766614 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" podUID="0f223a32-eac0-491e-ae99-0330a8043011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.783235 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.785303 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.184:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:28.785480 5043 logging.go:55] [core] [Channel #649 SubChannel #712]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.184:50052", ServerName: "10.132.0.184:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.184:50052: operation was canceled" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.787815 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.165:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.791280 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.174:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.888454 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.888803 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.896597 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.897175 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.181:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.897417 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.178:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:28.897612 5043 logging.go:55] [core] [Channel #648 SubChannel #717]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.174:50052", ServerName: "10.132.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.174:50052: operation was canceled" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.899217 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.968033756s: [/var/lib/containers/storage/overlay/ab6da4aae429b1bd88d53fb4faa8623ec593ef59b8a0a4c8b8fc52b085ea4020/diff /var/log/pods/openshift-ingress_e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87_842a6154-7c20-43e2-bc07-03a6cce2d71b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.905742 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.972859593s: [/var/lib/containers/storage/overlay/e117c61b4e02697e6ff05324fb75202cb6db15211303772110635999a03cc51a/diff /var/log/pods/e2e-models-e2e-worker-w1-04c5c2_simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj_a0f8db39-ca3d-4e82-a6de-774b40cb6325/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.916985 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.917264 5043 trace.go:236] Trace[1439034394]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" (31-Oct-2026 23:19:27.313) (total time: 1603ms): Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: Trace[1439034394]: [1.603341353s] [1.603341353s] END Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.925294 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.925886 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.177:50052\" within 1s: context deadline exceeded" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.941184 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.941224 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.941302 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:05.401442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:28.943010 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.014899 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.082247835s: [/var/lib/containers/storage/overlay/22af227a94096e199aad852b8f75fccb37d6d07cd99c73569ec18fba5c1cc6a9/diff /var/log/pods/openshift-ingress_e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml_b6a2e7de-778e-47c1-a35a-d4f066df6748/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.014951 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.014993 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.015454 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.015491 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.015619 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.015403 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.016096 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.057668 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.170:50052\" within 1s: context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.302123 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.169:50052\" within 1s: context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.303743 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.306754 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.308825 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.308869 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.309190 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.342700 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.175:50052\" within 1s: context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:29.342712 5043 logging.go:55] [core] [Channel #699 SubChannel #700]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.169:50052", ServerName: "10.132.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:60470->10.132.0.169:50052: use of closed network connection" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.344643 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.344651 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:19:29.347729 5043 logging.go:55] [core] [Channel #692 SubChannel #694]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:34852->10.132.0.59:50052: use of closed network connection" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.528221 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.528266 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.529336 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.570686 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.570684 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.574664 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.630666 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.632689 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.788143 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal" podUID="a70adfcc128189ec0c0f585313e4a644" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.906660 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.906718 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.925931 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:29.983682 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.012629 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.012633 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.012703 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.012669 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.385663 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.672667 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:30.926078 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:31.213024 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:31.294655 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:31.329037 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:31.412664 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:31.926236 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.550676 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.667588 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.737922 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.737978 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.754631 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.754674 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:32.926519 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:56.792512 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.063580 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.063628 5043 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.063580 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.063694 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7" podUID="0f223a32-eac0-491e-ae99-0330a8043011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.064083 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101210 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101272 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101324 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101342 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101367 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101600 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101817 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101839 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101864 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101877 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101897 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.101909 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102109 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102128 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102326 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102350 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102593 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102618 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102832 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.102851 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.192654 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.192752 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.192994 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.008299137s: [/var/lib/containers/storage/overlay/235339abbad24f749115edde190d818318b9154570e86451821a2231419157b9/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.441256 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeNotReady" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.656388 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.179927426s: [/var/lib/containers/storage/overlay/08ba556752740a1b0a1960f600ffc8e188a8853ddca3e5a3a8e2414353a46e2d/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.656421 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.063599 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.683719 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.683766 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:19:57.063628 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.684241 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.684491 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.604422771s: [/var/lib/containers/storage/overlay/569d076d942cf2ac59e91a81f63a5884f932ee43b71c4433437aedd74400e4d0/diff /var/log/pods/openshift-ingress_e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6_433ee218-d3e7-4837-98f8-642a6266fdf9/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.684521 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.435673706s: [/var/lib/containers/storage/overlay/2e67dc80796edcb1a4433af93aabbf87e8ad7a81e7e1da04d4fddd8a35905a55/diff /var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.698795 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.171:50052\" within 1s: context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.700472 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.701922 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.702260 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.702509 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.702842 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.707431 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.712329 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.41:8081/healthz\": dial tcp 10.132.0.41:8081: connect: connection reset by peer (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.712381 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/healthz\": dial tcp 10.132.0.41:8081: connect: connection reset by peer (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.714193 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.714827 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.719363 5043 trace.go:236] Trace[947597354]: "Calculate volume metrics of istio-token for pod openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" (31-Oct-2026 23:19:57.239) (total time: 25480ms): Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: Trace[947597354]: [25.48024221s] [25.48024221s] END Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.720257 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.720317 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.720648 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.720691 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.720899 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-wtb2p" podUID="756d759c-3587-4ff9-9abb-f24c46f64077" containerName="extractor" probeResult="failure" output="command timed out" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.723223 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.723621 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.724571 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.724624 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.726402 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.726704 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.728513 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.728580 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.729214 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.729923 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.730197 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.730491 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.732908 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.736277 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.736311 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/healthz\": context deadline exceeded" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.738249 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.767952 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.768035 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.791599 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.791667 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.793111 5043 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xbgvl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.151:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.793166 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.151:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.797636 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.797690 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused (Client.Timeout exceeded while awaiting headers)" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.800621 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.59:50052: i/o timeout" Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.801004 5043 trace.go:236] Trace[198683521]: "Calculate volume metrics of kube-api-access-gdlrr for pod openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-tzkd9" (31-Oct-2026 23:20:21.715) (total time: 1085ms): Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: Trace[198683521]: [1.085135534s] [1.085135534s] END Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.801067 5043 trace.go:236] Trace[173678702]: "Calculate volume metrics of model-cache for pod e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" (31-Oct-2026 23:20:21.715) (total time: 1085ms): Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: Trace[173678702]: [1.085206717s] [1.085206717s] END Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.801759 5043 trace.go:236] Trace[79905737]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" (31-Oct-2026 23:20:21.718) (total time: 1083ms): Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: Trace[79905737]: [1.083013333s] [1.083013333s] END Oct 31 23:20:30.869371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.819996 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.751174373s: [/var/lib/containers/storage/overlay/8332729cef23c13ef23161d99f432a616b25b53d589a995fd0db8570839d1ba4/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-ftgld_0063f042-8457-40ab-97d7-94d1420af56b/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.820055 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.571152889s: [/var/lib/containers/storage/overlay/c327292a7ee3ddf9914749a3378e9e0f92c75ceeac6df68311f77964c2b32776/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ttq5f_c8f7392b-faf8-43ad-a96b-7e95a95d7792/migrator/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.820080 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.571177953s: [/var/lib/containers/storage/overlay/00c6905b3898b1f788bc1f5f6fb029b1c0a4decd971c65cbc7d727628dde6f9d/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-gftjj_3ce54792-3b37-4895-a39a-d233a7e3d4ee/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.824841 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.830120 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.177:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.830196 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.170:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.830250 5043 trace.go:236] Trace[1662993329]: "Calculate volume metrics of workload-certs for pod openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" (31-Oct-2026 23:20:21.733) (total time: 1096ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[1662993329]: [1.096359113s] [1.096359113s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.831681 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.35522915s: [/var/lib/containers/storage/overlay/8ae55f810dd9d241f9b2438bc1bb8fa1981ee30ee18f36dd2ff5528f9f53d38d/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-khbjk_6c14a7d4-f9a1-4e58-8ee5-8455aecf175e/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.833775 5043 logging.go:55] [core] [Channel #744 SubChannel #754]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.177:50052", ServerName: "10.132.0.177:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.177:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.833861 5043 trace.go:236] Trace[270451920]: "Calculate volume metrics of manager-config for pod openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" (31-Oct-2026 23:19:57.072) (total time: 25761ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[270451920]: [25.761787309s] [25.761787309s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.834289 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.834738 5043 setters.go:546] "Node became not ready" node="ip-10-0-133-222.us-east-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T23:19:32Z","lastTransitionTime":"2026-10-31T23:19:32Z","reason":"KubeletNotReady","message":"container runtime is down"} Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.835509 5043 logging.go:55] [core] [Channel #745 SubChannel #755]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.175:50052", ServerName: "10.132.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.175:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.836124 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.836161 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.838233 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.192806 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.008112008s: [/var/lib/containers/storage/overlay/696f88b6b0bb5f7e2fadb32007118886ea374537ec0b51b666ac5c996b07b198/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.839134 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.171:50052: i/o timeout" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.839397 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.839777 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.175:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.840457 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.646789409s: [/var/lib/containers/storage/overlay/1de68f38afc1d163888e1d88fe5c1ef7fe3762d8d63fa14d27d8d4bcf5d2b1e5/diff /var/log/pods/openshift-multus_network-metrics-daemon-cz8ch_6c707771-1798-4442-9277-cecbc002c007/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.840497 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.364042608s: [/var/lib/containers/storage/overlay/633d15a8f9b53d7fde6ce446bc3e071d1cc36c3314a06c39cd87a20e8a7e33a7/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5blk9_b8edf410-4aa2-42ea-bf3f-016002b74ffa/8219b316cb51548d62306fbdb21ddfc5cf3cb49e2501d77ad8f697fab4704820.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.840585 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.646755909s: [/var/lib/containers/storage/overlay/8cd241bb780febd997d484fae83c2b0287190a56bfd76a60f6582c164a18eaed/diff /var/log/pods/openshift-ingress_e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg_d0b6666c-4eae-49da-9e9b-d2806ae3ed50/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.840628 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.62493341s: [/var/lib/containers/storage/overlay/5a5c20285656ab64e07ed40aff5122ce9ebf218c4fb3c3b78c5c0fd332656ce4/diff /var/log/pods/openshift-insights_insights-runtime-extractor-wtb2p_756d759c-3587-4ff9-9abb-f24c46f64077/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.840658 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.747749669s: [/var/lib/containers/storage/overlay/b65fd04db44ea1b2308c9ad93121d7684869baacdacea777286837dd8d6d39d9/diff /var/log/pods/e2e-models-e2e-worker-w0-a16dfb_distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m_3142d597-b170-4c68-8b9d-869fb8bda3b8/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.840684 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.752685556s: [/var/lib/containers/storage/overlay/b197943bc0cbf1a7cee8cff24c72d93c46b0ae62cedf7998e3d8ff54faf99e03/diff /var/log/pods/e2e-models-e2e-worker-w5-2294a7_premium-w5-2294a7-kserve-6d7b7f4f94-8jbls_f23336de-2435-41dd-a86e-629d91536e27/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841045 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.172:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.850900 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.165:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841248 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.851093 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841669 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.847249 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.842864 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.848992 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-jpmrk" podUID="5a940d2c-fc2c-40ae-a8bd-5916089eb925" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841492 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.852262 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.845844 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841685 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.852344 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.843613 5043 logging.go:55] [core] [Channel #748 SubChannel #752]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.170:50052", ServerName: "10.132.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.170:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.846037 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.852401 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.846098 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.852452 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.846131 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.852507 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.847498 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.859876 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.860887 5043 logging.go:55] [core] [Channel #746 SubChannel #770]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.165:50052", ServerName: "10.132.0.165:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.165:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.859118 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.847560 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.862371 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.862895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.848051 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.67656226s: [/var/lib/containers/storage/overlay/76b57e901a1f9e25431c9dfe6195ea37a9debfead22be15671dfa2f2589221e9/diff /var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-5blk9_b8edf410-4aa2-42ea-bf3f-016002b74ffa/cluster-samples-operator/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841615 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.863095 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.841108 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.863331 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.849913 5043 logging.go:55] [core] [Channel #749 SubChannel #762]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.172:50052", ServerName: "10.132.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.172:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.852911 5043 logging.go:55] [core] [Channel #750 SubChannel #753]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.171:50052", ServerName: "10.132.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.171:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.857333 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 30.001485988s: [/var/lib/containers/storage/overlay/04d4852636ef2e909efc7a085c5ceb371895f810a75ff879577872dcb1b04088/diff /var/log/pods/e2e-models-e2e-worker-w1-04c5c2_distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5_01442497-4a44-42db-bd0d-7d7839a7efe1/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.863738 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.174:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.867322 5043 logging.go:55] [core] [Channel #759 SubChannel #763]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.59:50052", ServerName: "10.132.0.59:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.59:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.869701 5043 logging.go:55] [core] [Channel #747 SubChannel #765]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.174:50052", ServerName: "10.132.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.174:50052: operation was canceled" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872122 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.109721443s: [/var/lib/containers/storage/overlay/f81344a15ed4a9d608810a33594805df2913e0893f78e84a239b39e7d5fcbc41/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872315 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.623414209s: [/var/lib/containers/storage/overlay/2fbb722fdd3f5970a56485eebd4b848d7b40ef2b67e73d318430573b19793882/diff /var/log/pods/openshift-ingress-canary_ingress-canary-r2hnp_f9191bfb-6d65-4d74-8948-fcdb9ad57b76/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872339 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.253324723s: [/var/lib/containers/storage/overlay/4baf6b9b52681bc7a1f12ea6b20809f044a5980da77387203b254d13a74456b8/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml_65dd98b9-8631-405f-88bc-f7a4e090177a/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872357 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.676105582s: [/var/lib/containers/storage/overlay/eacdf5971421dae26dd099e421030cbff79986460eede681550f33bf03d1c1e0/diff /var/log/pods/openshift-ingress_e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd_51479f0b-8c1b-472e-b4aa-80a088062549/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872477 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.67793834s: [/var/lib/containers/storage/overlay/22660c1b1bf6a129aea54b8efcd70cb912bf893119cf3fe60b63fe23d356d9d6/diff /var/log/pods/openshift-ingress_e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp_b2d8b699-311d-48f7-be78-2223ba357359/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872518 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.67352467s: [/var/lib/containers/storage/overlay/e126ec7fd5260905d7532ee9620a44f179949262daca0f5a4399436124f35c3b/diff /var/log/pods/kuadrant-system_authorino-6bfb984676-kv9dg_d55a9c0b-0e57-4c80-b2e2-929b74863635/authorino/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872538 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.39609106s: [/var/lib/containers/storage/overlay/9c6a701a6d1afc286a4ec76c3a654900938710819e1e9243ed79d33f187e045c/diff /var/log/pods/openshift-image-registry_node-ca-58lh2_37b0f64a-6235-4472-b7e2-a2d2a08fd388/node-ca/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872584 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.672386726s: [/var/lib/containers/storage/overlay/bcae4b22d3b5527388fef5ab9bf8ef860cd9eac4bf46536947048a3446a9ac69/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz_da7b510a-9644-48e0-96fc-29447a106ee5/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872626 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.797295182s: [/var/lib/containers/storage/overlay/ace5c054925273f5dc8469dcbb22ed8d35d12e777ef6bde5660cf89f777e0f0d/diff /var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-cbfsw_90d658ad-d248-4853-8bd8-b64076ec9968/check-endpoints/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872648 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.668699118s: [/var/lib/containers/storage/overlay/e1fb5fab3f59636add8fdd4040f4f66e54572259b34fb4019e9f13b0281a7ac3/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4_552eb1a2-3e7e-4311-add3-9dba9f0416af/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.872669 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.676181971s: [/var/lib/containers/storage/overlay/79cd714a7a294f16e9e3dbbb447d3334ddd0afb90968ec4a013e4d4f9f282119/diff /var/log/pods/openshift-ingress_e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf_3a9dcb05-0ae4-4b58-bed7-39549e5800cd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.874433 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podUID="8a1ca4ab-6864-49ea-af42-3c31f7823285" containerName="ovnkube-controller" probeResult="failure" output="command timed out" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.875398 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podUID="8a1ca4ab-6864-49ea-af42-3c31f7823285" containerName="sbdb" probeResult="failure" output="command timed out" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.875646 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podUID="8a1ca4ab-6864-49ea-af42-3c31f7823285" containerName="nbdb" probeResult="failure" output="command timed out" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.875738 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.880596 5043 trace.go:236] Trace[1288187226]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" (31-Oct-2026 23:19:35.376) (total time: 47504ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[1288187226]: [47.504317827s] [47.504317827s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.896455 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.898704 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.185:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.936681 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.936744 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.937659 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.938647 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.940620 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:22.940701 5043 logging.go:55] [core] [Channel #761 SubChannel #769]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.185:50052", ServerName: "10.132.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52792->10.132.0.185:50052: use of closed network connection" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.955970 5043 trace.go:236] Trace[192755206]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" (31-Oct-2026 23:19:55.264) (total time: 27691ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[192755206]: [27.691437606s] [27.691437606s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.974344 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.747938144s: [/var/lib/containers/storage/overlay/95c01fb5e39c8cd6691774bf2240f8e5419cfbf0e7d7366399caa9f3591e2240/diff /var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-8g88j_1d95fc77-4bb6-4464-97f2-2bd5500d88e1/service-ca-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.974407 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.76226014s: [/var/lib/containers/storage/overlay/118553b56496c8bc700d3bb067030dfd1eef18c497b4913525010dcaebf75a43/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6f9f_8a1ca4ab-6864-49ea-af42-3c31f7823285/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:22.979619 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.20377841s: [/var/lib/containers/storage/overlay/8b6116cd674a5f281ecc8b2d6d7821eadf35ba8cac698111f91f598b46e7af34/diff /var/log/pods/openshift-ingress_e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs_4ce8af1c-a77d-44b2-920d-9d57b20a21ad/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.097946 5043 trace.go:236] Trace[940145106]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" (31-Oct-2026 23:19:57.192) (total time: 25905ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[940145106]: [25.905064089s] [25.905064089s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.157351 5043 trace.go:236] Trace[1566772551]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" (31-Oct-2026 23:19:57.238) (total time: 25918ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[1566772551]: [25.918843069s] [25.918843069s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.191358 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.191603 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.194697 5043 trace.go:236] Trace[1916675010]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" (31-Oct-2026 23:19:57.065) (total time: 26129ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[1916675010]: [26.129471255s] [26.129471255s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.199915 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.184:50052\" within 1s: context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.202967 5043 trace.go:236] Trace[1459357998]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" (31-Oct-2026 23:19:57.076) (total time: 26126ms): Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: Trace[1459357998]: [26.126689793s] [26.126689793s] END Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:23.231699 5043 logging.go:55] [core] [Channel #751 SubChannel #757]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.164:50052", ServerName: "10.132.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:50128->10.132.0.164:50052: use of closed network connection" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.234645 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:23.240690 5043 logging.go:55] [core] [Channel #756 SubChannel #758]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.184:50052", ServerName: "10.132.0.184:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47666->10.132.0.184:50052: use of closed network connection" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.282104 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 26.195899551s: [/var/lib/containers/storage/overlay/bab297c205320506e00e3f6eec994a4a4c16c3429e36536ab392b2d8806481d0/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx_1d56bf0b-7968-4952-9917-ba0bc5cba955/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.379700 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 26.194495162s: [/var/lib/containers/storage/overlay/0a75e72bf837ed81d442a15707614f1f80f3df7d574baf725ecdcb28f89bf6f0/diff /var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-qv9l7_eec52f11-3785-48d2-92a1-cbd30a6047e2/networking-console-plugin/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.458600 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.458661 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.465153 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.469664 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.471171 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.506656 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.656709 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.684141 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.702415 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.702441 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.702465 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.707696 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.713806 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.714921 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.714920 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.724264 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.724278 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.727414 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.727442 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.730583 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.730593 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.730631 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.769371 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.769449 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:06.099403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.774653 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.779636 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.793709 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.793775 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.181:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.834750 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:23.834774 5043 logging.go:55] [core] [Channel #760 SubChannel #766]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.181:50052", ServerName: "10.132.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52438->10.132.0.181:50052: use of closed network connection" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.840241 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.851686 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.851692 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.165:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.851973 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.861598 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.869832 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.174:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.875655 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.875694 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:23.892700 5043 logging.go:55] [core] [Channel #771 SubChannel #774]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.165:50052", ServerName: "10.132.0.165:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:45488->10.132.0.165:50052: use of closed network connection" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.897169 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.899343 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.185:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:23.910711 5043 logging.go:55] [core] [Channel #778 SubChannel #779]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.174:50052", ServerName: "10.132.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:50702->10.132.0.174:50052: use of closed network connection" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.916668 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:23.939772 5043 logging.go:55] [core] [Channel #781 SubChannel #782]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.185:50052", ServerName: "10.132.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52820->10.132.0.185:50052: use of closed network connection" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:23.980676 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:24.192127 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:24.192229 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:24.200375 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.184:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:24.232735 5043 logging.go:55] [core] [Channel #784 SubChannel #785]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.164:50052", ServerName: "10.132.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:50152->10.132.0.164:50052: use of closed network connection" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.138913 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.139134 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.139220 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.139528 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.139577 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.139677 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.159286 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.258885 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.286924 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.287090 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.305275 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.305838 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.305881 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.376208 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.376257 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.376328 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.376371 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.881748 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.911573 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.716337258s: [/var/lib/containers/storage/overlay/646bdbbb0a8b087dcc8b8524cd69d6d027c2641231aef585436c49fa2990bdb9/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-64f549b4b7-q5w9p_863f9cb3-dbae-4b56-b240-eb661de1f2dc/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.917224 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.917276 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.918564 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.918608 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.941684 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:25.972124 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.042236 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.132282431s: [/var/lib/containers/storage/overlay/7e9169b92994b737de5aaa4b48a0b479232f889cb151365053f775a597418c71/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.052888 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.851127086s: [/var/lib/containers/storage/overlay/17b3c5900e84ff56a24798f73e9ebccd636150ebc8e3d268558ccef660ac1f1f/diff /var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.061701 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.061738 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.380716 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.18253353s: [/var/lib/containers/storage/overlay/be4d3d14d0f231879ef24083b7b2547029946e6eaead33e3fb15d4e7421b6cd3/diff /var/log/pods/openshift-ingress_e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf_cd24c44a-38fc-460e-a833-fd5308961380/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.387926 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.390167 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.178:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.390771 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.182:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:26.390919 5043 logging.go:55] [core] [Channel #799 SubChannel #801]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.178:50052", ServerName: "10.132.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.178:50052: operation was canceled" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.391472 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.169:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:26.391601 5043 logging.go:55] [core] [Channel #800 SubChannel #803]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.182:50052", ServerName: "10.132.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.182:50052: operation was canceled" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:26.392137 5043 logging.go:55] [core] [Channel #802 SubChannel #804]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.169:50052", ServerName: "10.132.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.169:50052: operation was canceled" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.392723 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.392753 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.407902 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.11:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.407937 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.478437 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:26.478472 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:20:26.496903 5043 logging.go:55] [core] [Channel #787 SubChannel #788]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.184:50052", ServerName: "10.132.0.184:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47686->10.132.0.184:50052: use of closed network connection" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.903182 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.903999 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.327832073s: [/var/lib/containers/storage/overlay/fef03606fe9481f7d44b2c8982b93ee22662771017ace209589d27a7850366da/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-jqhtb_1d96328a-6b55-4673-9ad7-480e8b36dc39/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.904326 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.341194374s: [/var/lib/containers/storage/overlay/31ad3073a259c11b15f33570496a1821e46de62a092457b7fd10b55b53502949/diff /var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ld9pm_98438769-91a9-4aec-b9e9-d0c4044cae37/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.935147 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.935249 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.940789 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.944184 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-jpmrk" podUID="5a940d2c-fc2c-40ae-a8bd-5916089eb925" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.947211 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.947286 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.947315 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:27.947739 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-jpmrk" podUID="5a940d2c-fc2c-40ae-a8bd-5916089eb925" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:28.898702 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:28.903676 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:28.904433 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:28.979694 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:28.981214 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:28.981265 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.053122 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.053190 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.106637 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.257517 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": context deadline exceeded" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.257725 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.567818 5043 trace.go:236] Trace[314086837]: "iptables ChainExists" (31-Oct-2026 23:19:57.190) (total time: 32377ms): Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: Trace[314086837]: [32.377336508s] [32.377336508s] END Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:29.654801 5043 trace.go:236] Trace[1225919453]: "iptables ChainExists" (31-Oct-2026 23:19:57.141) (total time: 32513ms): Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: Trace[1225919453]: [32.513537142s] [32.513537142s] END Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:30.859053 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:20:30.872994 5043 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.508703097s: [/var/lib/containers/storage/overlay/7e57372aefe82fc3729cfc61e73f811a01262a24a9f31c389dcdd3aedb0c4b23/diff /var/log/pods/openshift-ingress_e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb_d336aa46-fc0a-4c34-a9dc-a45949939314/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.273822 5043 ???:1] "http: TLS handshake error from 10.0.131.157:50370: EOF" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.273865 5043 scope.go:122] "RemoveContainer" containerID="df4706d9b3560b24f8d925e2deeeb5948c921ee75380b3f7bb8f8e64daca8b4d" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.274310 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-wtb2p" podUID="756d759c-3587-4ff9-9abb-f24c46f64077" containerName="extractor" probeResult="failure" output="command timed out" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:06.290436 5043 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:06.290466 5043 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:06.290718 5043 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:06.290730 5043 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.291504 5043 patch_prober.go:29] interesting pod/dns-default-xctjg container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.291534 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-xctjg" podUID="67c4f288-6571-456f-8aa8-d6f5947eb14d" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.21:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.294180 5043 ???:1] "http: TLS handshake error from 10.0.131.157:41831: EOF" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.294494 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podUID="8a1ca4ab-6864-49ea-af42-3c31f7823285" containerName="ovnkube-controller" probeResult="failure" output="command timed out" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.294621 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podUID="8a1ca4ab-6864-49ea-af42-3c31f7823285" containerName="nbdb" probeResult="failure" output="command timed out" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.294703 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f6f9f" podUID="8a1ca4ab-6864-49ea-af42-3c31f7823285" containerName="sbdb" probeResult="failure" output="command timed out" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.295019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.296189 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.181:50052: i/o timeout" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.304479 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.304521 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.306737 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.307712 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.310842 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": dial tcp 10.132.0.40:8081: connect: connection refused" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.332343 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/healthz\": dial tcp 10.132.0.40:8081: connect: connection refused" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.334230 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": dial tcp 10.132.0.34:8081: connect: connection refused" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.351137 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.359820 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.377767 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.182:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:06.382088 5043 logging.go:55] [core] [Channel #819 SubChannel #821]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.182:50052", ServerName: "10.132.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.182:50052: operation was canceled" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.397803 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.164:50052\" within 1s: context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:06.398660 5043 logging.go:55] [core] [Channel #815 SubChannel #825]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.164:50052", ServerName: "10.132.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.164:50052: operation was canceled" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.409687 5043 patch_prober.go:29] interesting pod/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.409777 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.778157 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.778223 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.780707 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.781464 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.781498 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.787068 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.787107 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.793920 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.793984 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.800825 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.800858 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.809336 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.818873 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.819598 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:17.541276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.819644 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.838831 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.839694 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.839743 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.840148 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.840186 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.842683 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.842723 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.850510 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:06.850607 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.201921 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.165:50052\" within 1s: context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.205141 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.205141 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.206315 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.181:50052\" within 1s: context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.206421 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.206455 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:07.242703 5043 logging.go:55] [core] [Channel #816 SubChannel #828]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.165:50052", ServerName: "10.132.0.165:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52606->10.132.0.165:50052: use of closed network connection" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:07.246686 5043 logging.go:55] [core] [Channel #826 SubChannel #827]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.181:50052", ServerName: "10.132.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:49418->10.132.0.181:50052: use of closed network connection" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.247691 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.265017 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.265147 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.270708 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.273129 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.285341 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" podUID="f091b8de-14bb-4d9d-ac75-3edd31cb3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.299397 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.184:50052\" within 1s: context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.301438 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.301455 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.304811 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.308218 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.314535 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.185:50052\" within 1s: context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.321791 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.321829 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:07.339702 5043 logging.go:55] [core] [Channel #812 SubChannel #813]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.184:50052", ServerName: "10.132.0.184:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:44162->10.132.0.184:50052: use of closed network connection" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.353636 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.353673 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:07.355698 5043 logging.go:55] [core] [Channel #814 SubChannel #817]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.185:50052", ServerName: "10.132.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:55192->10.132.0.185:50052: use of closed network connection" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.360235 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.360268 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.362649 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.362691 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.378419 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.182:50052\" within 1s: context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.391654 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.391662 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.391682 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:07.418708 5043 logging.go:55] [core] [Channel #822 SubChannel #823]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.182:50052", ServerName: "10.132.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:54174->10.132.0.182:50052: use of closed network connection" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.780884 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.796617 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.796880 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" podUID="98438769-91a9-4aec-b9e9-d0c4044cae37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.820080 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.841773 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.841979 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.842004 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.843046 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:07.843076 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:08.202521 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.165:50052\" within 1s: context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:08.203039 5043 logging.go:55] [core] [Channel #831 SubChannel #832]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.165:50052", ServerName: "10.132.0.165:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:52622->10.132.0.165:50052: use of closed network connection" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:08.249170 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:08.302521 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:08.308831 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:08.893656 5043 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-2tbhj container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:08.893720 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:09.256954 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:09.257009 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:09.309811 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:09.894750 5043 patch_prober.go:29] interesting pod/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:09.894813 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:10.212021 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": dial tcp 10.132.0.34:8081: connect: connection refused" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:10.266944 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:10.310499 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:10.537390 5043 trace.go:236] Trace[537155049]: "iptables ChainExists" (31-Oct-2026 23:21:05.782) (total time: 4755ms): Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: Trace[537155049]: [4.755067154s] [4.755067154s] END Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:10.537393 5043 trace.go:236] Trace[651683721]: "iptables ChainExists" (31-Oct-2026 23:21:05.793) (total time: 4743ms): Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: Trace[651683721]: [4.74378053s] [4.74378053s] END Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:11.311091 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:11.328538 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:11.330919 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" podUID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": dial tcp 10.132.0.40:8081: connect: connection refused" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:12.311740 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:12.669633 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:12.669689 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:12.669702 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:12.669715 5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.us-east-2.compute.internal" event="NodeReady" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.062614 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:13.197794 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1m47.145s" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.201154 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.201183 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.202464 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916"} pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.202529 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" containerID="cri-o://d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916" gracePeriod=10 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.207486 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": dial tcp 10.132.0.34:8081: connect: connection refused" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.221159 5043 generic.go:356] "Generic (PLEG): container finished" podID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerID="8802685cf1bf4a2939a3fd6d39db6ca13a83ed567d0f9ce0cffe32361a521f87" exitCode=1 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.248696 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"6034260101de910aa610eb5cd1a35ee8b59069b7b6b37f14544aaf6115255d9a"} pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.248752 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" containerID="cri-o://6034260101de910aa610eb5cd1a35ee8b59069b7b6b37f14544aaf6115255d9a" gracePeriod=10 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.249336 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"9a887c9eb4fd3578596aa13886ac4a95160e1df5231b036f194396c2cbc800d5"} pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" containerMessage="Container console-operator failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.249386 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" containerID="cri-o://9a887c9eb4fd3578596aa13886ac4a95160e1df5231b036f194396c2cbc800d5" gracePeriod=30 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.270810 5043 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" secret="" err="secret \"e2e-mt-85b90bac-openshift-default-dockercfg-lqfl8\" not found" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.271150 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="router" containerStatusID={"Type":"cri-o","ID":"5106193ed3e2ad49ca55a53cda62c6625603ac001a10b2f0d38402b7c1ee3049"} pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" containerMessage="Container router failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.271189 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" containerID="cri-o://5106193ed3e2ad49ca55a53cda62c6625603ac001a10b2f0d38402b7c1ee3049" gracePeriod=10 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.315815 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"0cd971070a0d1c8f922c607fc23f0dcfe7f7d5926e5ac627e4fedb9508f43045"} pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.315887 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" containerID="cri-o://0cd971070a0d1c8f922c607fc23f0dcfe7f7d5926e5ac627e4fedb9508f43045" gracePeriod=10 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.318588 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.376078 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"84bd33db9643630363cebf3fecad3b9b291905048da6f3b662d0d8deb5aa49fd"} pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.376144 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" podUID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerName="manager" containerID="cri-o://84bd33db9643630363cebf3fecad3b9b291905048da6f3b662d0d8deb5aa49fd" gracePeriod=10 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.380659 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"8802685cf1bf4a2939a3fd6d39db6ca13a83ed567d0f9ce0cffe32361a521f87"} pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.380711 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" containerID="cri-o://8802685cf1bf4a2939a3fd6d39db6ca13a83ed567d0f9ce0cffe32361a521f87" gracePeriod=10 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.404840 5043 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" secret="" err="secret \"payload-processing-e2e-mt-85b90bac-dockercfg-mfhg6\" not found" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.419494 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"e0bacb8fcf4bfc421543a85e4f60c9b4faecf99a9cc250b49b2f7f2455c8c902"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" containerMessage="Container csi-driver failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.419629 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" podUID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerName="csi-driver" containerID="cri-o://e0bacb8fcf4bfc421543a85e4f60c9b4faecf99a9cc250b49b2f7f2455c8c902" gracePeriod=30 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.460239 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.558018 5043 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="cert-manager-webhook" containerStatusID={"Type":"cri-o","ID":"df921a44d905ad3447310ca266e1269af283784ac8a88b4c5d237bc01349c26c"} pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" containerMessage="Container cert-manager-webhook failed liveness probe, will be restarted" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:13.558069 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" containerID="cri-o://df921a44d905ad3447310ca266e1269af283784ac8a88b4c5d237bc01349c26c" gracePeriod=30 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:13.570259 5043 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-85b90bac: configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:13.571070 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume podName:06197912-3b3e-40aa-a19b-8025350bf804 nodeName:}" failed. No retries permitted until 2026-10-31 23:21:14.071045989 +0000 UTC m=+1900.300600389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume") pod "payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" (UID: "06197912-3b3e-40aa-a19b-8025350bf804") : configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013563 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013630 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013649 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013676 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013690 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013717 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" event={"ID":"863f9cb3-dbae-4b56-b240-eb661de1f2dc","Type":"ContainerDied","Data":"8802685cf1bf4a2939a3fd6d39db6ca13a83ed567d0f9ce0cffe32361a521f87"} Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013748 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013776 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013795 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013813 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013837 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013851 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013871 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013887 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013941 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8da9f27-kx6jq"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.013987 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014008 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014036 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014058 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014103 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014121 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bcdb010-mpvjz"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014158 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014191 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014209 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27ecdf06-zhpzq"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014226 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014243 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86fa2ff4-m8lnh"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014260 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014288 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014332 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014364 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014379 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014641 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014713 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.014736 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09922503-rc6s5"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.015372 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.015398 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.017463 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.017501 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02ccdc3-wr76q"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.017899 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.017935 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.017957 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472a38b0-gmdg5"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.018164 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.018196 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l"] Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.032636 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.032677 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.033153 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.033183 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:14.153918 5043 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-85b90bac: configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:14.153990 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume podName:06197912-3b3e-40aa-a19b-8025350bf804 nodeName:}" failed. No retries permitted until 2026-10-31 23:21:15.153972656 +0000 UTC m=+1901.383527047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume") pod "payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" (UID: "06197912-3b3e-40aa-a19b-8025350bf804") : configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.211145 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.211251 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.239425 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.142903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.259954 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" containerID="cri-o://c80e9eb07b05228715ce9d5fb8f1ea4fc46c76023c0c99f5b2867ca1725362a9" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.260046 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" containerID="cri-o://35469d41d88c962f7545fe8df512a334b4e1c24bd26d8c77412b355fac00cf50" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.260201 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" containerID="cri-o://37102e32d4a7f763663cdf7c0120f8d4a6ac2dd5f783fab9ffd21b1f5b79ae2e" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.260362 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.261156 5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.443597 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.661721 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.852150 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996402 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996445 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66cc15d5-87j8v"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996494 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996514 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996532 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996573 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996600 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996671 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xctjg" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996694 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996715 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996729 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996747 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996765 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473e18ab-bhjbl"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996781 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996796 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0083906-znt5b"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996812 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996833 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996861 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996883 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996904 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996931 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996952 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996969 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.996986 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997001 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997021 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997044 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f08a863-5g4jw"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997082 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997098 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997130 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997144 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fd3132-v8mbl"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997175 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997189 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997204 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:14.997812 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.038990 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" containerID="cri-o://3d8fadd7f7cde659665f33db449490030d5933036832b9df1eab3dc50d79cd5e" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.058022 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.058138 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.058227 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.058311 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.072173 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" containerID="cri-o://d92b844ec596fa77e27176e042eca461aad29ad582463d39277820c3fc5f9018" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.087140 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.087280 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.087354 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5kg\" (UniqueName: \"kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.123789 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.123860 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="Get \"http://10.132.0.11:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.158009 5043 trace.go:236] Trace[205113484]: "Calculate volume metrics of manifests for pod opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" (31-Oct-2026 23:19:57.197) (total time: 77960ms): Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: Trace[205113484]: [1m17.960326856s] [1m17.960326856s] END Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.207302 5043 csi_client.go:182] "Error calling CSI NodeGetInfo()" err="rpc error: code = Unavailable desc = error reading from server: EOF" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:15.207331 5043 csi_node_updater.go:178] "Failed to update CSIDriver" err="failed to get NodeGetInfo from driver \"ebs.csi.aws.com\": rpc error: code = Unavailable desc = error reading from server: EOF" driver="ebs.csi.aws.com" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.275461 5043 generic.go:356] "Generic (PLEG): container finished" podID="677d053b-fbf4-4b76-9782-6b3235dc340f" containerID="6b4da7546ae8e32d93087373e955abf2d278c3d6b647180e4b05f83dc7b165ff" exitCode=1 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312476 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" event={"ID":"677d053b-fbf4-4b76-9782-6b3235dc340f","Type":"ContainerDied","Data":"6b4da7546ae8e32d93087373e955abf2d278c3d6b647180e4b05f83dc7b165ff"} Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312577 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312603 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312633 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312651 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312670 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec7f2fd-gw4bt"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312695 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312718 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312745 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312798 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.312812 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.314094 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" containerID="cri-o://4854b34e74d727bf0de49827be4ab51521d669dc74f5a4b16eaee49d3b7f1376" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.314265 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" containerID="cri-o://0031786746efc4e173db89f8d083137c09cae82b1884156665e1f8e327d516ab" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.318226 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.412077 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581227 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581291 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581312 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581337 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581376 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581443 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581460 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581484 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581508 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581534 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581594 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581622 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xctjg" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581646 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581672 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581697 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.581907 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.582632 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.582858 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.582905 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.582971 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583000 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583033 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583065 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583113 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583140 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583167 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583206 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583234 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583254 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583271 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583292 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583317 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583337 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583373 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583414 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583443 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583492 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583540 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583602 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-ld9pm" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583639 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.583656 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.598777 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.614783 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.615065 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-2859g" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.615126 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.615188 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.615235 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.615272 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jpmrk" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.620412 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.710937 5043 scope.go:122] "RemoveContainer" containerID="6b4da7546ae8e32d93087373e955abf2d278c3d6b647180e4b05f83dc7b165ff" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.718886 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.719180 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.719366 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5kg\" (UniqueName: \"kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:15.720074 5043 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-85b90bac: configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:15.720144 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume podName:06197912-3b3e-40aa-a19b-8025350bf804 nodeName:}" failed. No retries permitted until 2026-10-31 23:21:17.720126788 +0000 UTC m=+1903.949681192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume") pod "payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" (UID: "06197912-3b3e-40aa-a19b-8025350bf804") : configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.784079 5043 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" secret="" err="secret \"payload-processing-e2e-mt-85b90bac-dockercfg-mfhg6\" not found" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.821046 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.821185 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72xl\" (UniqueName: \"kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.821427 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.825594 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: [+]has-synced ok Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: [-]process-running failed: reason withheld Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: healthz check failed Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.825639 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.924442 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.924578 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.925073 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klr5g\" (UniqueName: \"kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.925668 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d72xl\" (UniqueName: \"kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.926191 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:15.926322 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.034541 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klr5g\" (UniqueName: \"kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.039066 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.039416 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.044683 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.081650 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.081716 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.272804 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.454110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.779033 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ded6680-9832-4e62-b762-f0665c04996a" path="/var/lib/kubelet/pods/1ded6680-9832-4e62-b762-f0665c04996a/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.793519 5043 generic.go:356] "Generic (PLEG): container finished" podID="b4fd03ee-af94-4df1-b834-eced676d664e" containerID="6034260101de910aa610eb5cd1a35ee8b59069b7b6b37f14544aaf6115255d9a" exitCode=1 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:16.816071 5043 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" secret="" err="secret \"payload-processing-e2e-mt-85b90bac-dockercfg-mfhg6\" not found" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.068994 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.112930 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.112986 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:17.287512 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.297263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.328818 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:17.329018 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.404215 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d68379-d5b2-4caf-af89-9a293fea71b7" path="/var/lib/kubelet/pods/28d68379-d5b2-4caf-af89-9a293fea71b7/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.404489 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.436338 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.467831 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33aae16c-c087-4ef8-8d50-da1bcaab927a" path="/var/lib/kubelet/pods/33aae16c-c087-4ef8-8d50-da1bcaab927a/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.481109 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.485735 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.486194 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.543573 5043 trace.go:236] Trace[344787993]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" (31-Oct-2026 23:19:32.239) (total time: 105304ms): Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: Trace[344787993]: [1m45.304179015s] [1m45.304179015s] END Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.546635 5043 scope.go:122] "RemoveContainer" containerID="97a6008ba5f7ac6dce263300b8d35dfaf62abe78dc52aeeb4e82f00b3f351c06" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.550045 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a714fb-e0bb-45e8-a5d2-5ea07beb583f" path="/var/lib/kubelet/pods/37a714fb-e0bb-45e8-a5d2-5ea07beb583f/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.577941 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.577967 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.579677 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.606827 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c58484-9578-4663-813f-ae4b5c8490f7" path="/var/lib/kubelet/pods/52c58484-9578-4663-813f-ae4b5c8490f7/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.638924 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a5b524-ad27-4467-8e8d-4f57fdc73db2" path="/var/lib/kubelet/pods/56a5b524-ad27-4467-8e8d-4f57fdc73db2/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.654114 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c1b48b-4e73-4d61-bd7a-36689a3a06fe" path="/var/lib/kubelet/pods/60c1b48b-4e73-4d61-bd7a-36689a3a06fe/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.677531 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8998023d-f6f6-4afa-aa08-c09d4d371ff4" path="/var/lib/kubelet/pods/8998023d-f6f6-4afa-aa08-c09d4d371ff4/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.689145 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.689466 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klr5g\" (UniqueName: \"kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g\") pod \"maas-api-revoke-keys-e2e-mt-36a15fc5-twvct\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.698753 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf025a7-87b6-4a6c-9a76-44e61035e408" path="/var/lib/kubelet/pods/8bf025a7-87b6-4a6c-9a76-44e61035e408/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.699743 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b9fba4-ce9b-4012-a45d-ce2bb428079c" path="/var/lib/kubelet/pods/a0b9fba4-ce9b-4012-a45d-ce2bb428079c/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.709401 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93f9a34-e282-404d-b2b2-3374cdedbd6e" path="/var/lib/kubelet/pods/a93f9a34-e282-404d-b2b2-3374cdedbd6e/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.710031 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af858a4d-b7eb-4d98-bba4-8e29f15b6791" path="/var/lib/kubelet/pods/af858a4d-b7eb-4d98-bba4-8e29f15b6791/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.734856 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.766405 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.766888 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.768427 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.769019 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5kg\" (UniqueName: \"kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.769365 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.769417 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:17.770466 5043 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-85b90bac: configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:17.770528 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume podName:06197912-3b3e-40aa-a19b-8025350bf804 nodeName:}" failed. No retries permitted until 2026-10-31 23:21:21.770512602 +0000 UTC m=+1908.000067004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume") pod "payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" (UID: "06197912-3b3e-40aa-a19b-8025350bf804") : configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.781895 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47c9543-dee3-4e8e-8037-6dea7fba33f6" path="/var/lib/kubelet/pods/f47c9543-dee3-4e8e-8037-6dea7fba33f6/volumes" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:17.782355 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.854s" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782375 5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782408 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" event={"ID":"b4fd03ee-af94-4df1-b834-eced676d664e","Type":"ContainerDied","Data":"6034260101de910aa610eb5cd1a35ee8b59069b7b6b37f14544aaf6115255d9a"} Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782526 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782568 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782592 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782617 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782645 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782736 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782759 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782777 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782790 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782808 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782820 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782833 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct"] Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782844 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" containerID="cri-o://b338d82c0b906c30d6ac55ae97fc8626e6df193597e361a5bfe39359cf9c4f11" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.782952 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" containerID="cri-o://a0adc13a333dfdbba56b1a5e29430c78284ab010e8f20e3ad316f69d16f26947" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.783032 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" containerID="cri-o://e29ef966fe50e855687948706d2a78d30d1f4e59383978322d4d6003ad55b408" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.783252 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" containerID="cri-o://1e3c5dd51bbb61614ccbdce07be2fbcbd5083fc2914e57121c3ec8733122feac" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.783301 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerName="istio-proxy" containerID="cri-o://a112283e701bccf3ade34448bb910c025ecbd7961915ee71787f35773df4709c" gracePeriod=30 Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.808172 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:21:18.899522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.808213 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.809089 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.809121 5043 generic.go:356] "Generic (PLEG): container finished" podID="c8c42546-d756-4965-96c3-e3e05dd3d89b" containerID="e0bacb8fcf4bfc421543a85e4f60c9b4faecf99a9cc250b49b2f7f2455c8c902" exitCode=2 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.809192 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" event={"ID":"c8c42546-d756-4965-96c3-e3e05dd3d89b","Type":"ContainerDied","Data":"e0bacb8fcf4bfc421543a85e4f60c9b4faecf99a9cc250b49b2f7f2455c8c902"} Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.811608 5043 generic.go:356] "Generic (PLEG): container finished" podID="485a8e4b-a74c-4463-affc-44a6374fb81c" containerID="84bd33db9643630363cebf3fecad3b9b291905048da6f3b662d0d8deb5aa49fd" exitCode=0 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.811668 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" event={"ID":"485a8e4b-a74c-4463-affc-44a6374fb81c","Type":"ContainerDied","Data":"84bd33db9643630363cebf3fecad3b9b291905048da6f3b662d0d8deb5aa49fd"} Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.815126 5043 generic.go:356] "Generic (PLEG): container finished" podID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerID="d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916" exitCode=0 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.815210 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" event={"ID":"e0a050c1-60b2-4ee9-a860-8f646427645e","Type":"ContainerDied","Data":"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916"} Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.830594 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.832848 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72xl\" (UniqueName: \"kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl\") pod \"maas-api-revoke-sub-ab1ad7fa-wsj79\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.857008 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:17.857059 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.011294 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr"] Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.051949 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.395308 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj"] Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.395397 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.399728 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4eff5f-dockercfg-mqxft\"" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.401192 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4eff5f\"" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.486229 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsbs\" (UniqueName: \"kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs\") pod \"payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.486282 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.526071 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559776 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/1.log" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559808 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj"] Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559815 5043 generic.go:356] "Generic (PLEG): container finished" podID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerID="9a887c9eb4fd3578596aa13886ac4a95160e1df5231b036f194396c2cbc800d5" exitCode=1 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559841 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559848 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" event={"ID":"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5","Type":"ContainerDied","Data":"9a887c9eb4fd3578596aa13886ac4a95160e1df5231b036f194396c2cbc800d5"} Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559885 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr"] Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.559952 5043 scope.go:122] "RemoveContainer" containerID="b5633aa8e14dad2904bb2a51b1504d7aee23ca62f5979a6e2a226be181bd8bb5" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.560195 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf"] Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.564417 5043 generic.go:356] "Generic (PLEG): container finished" podID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerID="df921a44d905ad3447310ca266e1269af283784ac8a88b4c5d237bc01349c26c" exitCode=0 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.590156 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqzp\" (UniqueName: \"kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp\") pod \"payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.590458 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.590525 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsbs\" (UniqueName: \"kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs\") pod \"payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.590579 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.672066 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.691792 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqzp\" (UniqueName: \"kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp\") pod \"payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.692040 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.709348 5043 scope.go:122] "RemoveContainer" containerID="4895b295bd8fff51fb6f4fcc2c8b9bcb381f4720616b5c860d2558454f5728fa" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.709360 5043 generic.go:356] "Generic (PLEG): container finished" podID="0063f042-8457-40ab-97d7-94d1420af56b" containerID="0cd971070a0d1c8f922c607fc23f0dcfe7f7d5926e5ac627e4fedb9508f43045" exitCode=0 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.746315 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.746421 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf"] Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.746510 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" event={"ID":"cdefa176-e223-4de2-857d-14d6ffecf8a7","Type":"ContainerDied","Data":"df921a44d905ad3447310ca266e1269af283784ac8a88b4c5d237bc01349c26c"} Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.746537 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.746855 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqzp\" (UniqueName: \"kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp\") pod \"payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.746558 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" event={"ID":"0063f042-8457-40ab-97d7-94d1420af56b","Type":"ContainerDied","Data":"0cd971070a0d1c8f922c607fc23f0dcfe7f7d5926e5ac627e4fedb9508f43045"} Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.751649 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ba9088-openshift-default-dockercfg-zknlg\"" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794354 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794407 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79mj\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794610 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794666 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794766 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794933 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794961 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.794984 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.795083 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.802791 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.802830 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.815798 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsbs\" (UniqueName: \"kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs\") pod \"payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903111 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903178 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79mj\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903336 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903386 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903437 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903571 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903599 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903623 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.903693 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.906528 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.907192 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.907732 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.911617 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.911754 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.923043 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.936717 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.937076 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:18.984921 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.040017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.008953 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:19.201482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.161203 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.201482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.161209 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79mj\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj\") pod \"e2e-api-b-ba9088-openshift-default-6869694875-6klsf\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.201482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.196114 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:19.786777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.638912 5043 scope.go:122] "RemoveContainer" containerID="733d30cfc50d6e923776884427989cfcfa602a668bd2d2474fad961001f43169" Oct 31 23:21:19.786777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.641419 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:19.957719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:19.859891 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:20.217396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:20.217359 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:20.294355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:20.294270 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" podUID="cdefa176-e223-4de2-857d-14d6ffecf8a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": dial tcp 10.132.0.37:6080: connect: connection refused" Oct 31 23:21:20.844884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:20.769866 5043 scope.go:122] "RemoveContainer" containerID="ec916a0a1c9298d580dca1b7bc46bade1aa23c823e05ade58376020fa021aeae" Oct 31 23:21:20.985403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:20.951578 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s"] Oct 31 23:21:20.985403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:20.958821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:21.212298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.186380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:21.212298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.204749 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d"] Oct 31 23:21:21.215814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.215792 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.297161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.295877 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-885fdf\"" Oct 31 23:21:21.297161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.296779 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s"] Oct 31 23:21:21.297161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.296825 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d"] Oct 31 23:21:21.297161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.296850 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l"] Oct 31 23:21:21.297161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.296951 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.322938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.309058 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgrl\" (UniqueName: \"kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl\") pod \"payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.322938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.309134 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxrl\" (UniqueName: \"kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl\") pod \"payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.322938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.309386 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.322938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.309520 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.363047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.327800 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-885fdf-dockercfg-qxxpj\"" Oct 31 23:21:21.363047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.330130 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.411343 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.411479 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgrl\" (UniqueName: \"kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl\") pod \"payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.411524 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxrl\" (UniqueName: \"kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl\") pod \"payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.411803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.412352 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.412757 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.431480 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxrl\" (UniqueName: \"kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl\") pod \"payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:21.432531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.431680 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgrl\" (UniqueName: \"kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl\") pod \"payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:21.646734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.575311 5043 scope.go:122] "RemoveContainer" containerID="7720ba9aad150783629fecdc488a015f7396ceb840a97b36fdc8237c5f9f247f" Oct 31 23:21:21.728324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.662726 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:21:21.728324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.662789 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:21:21.738086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.738008 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:21.861085 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:21.821984 5043 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-85b90bac: configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:21.861085 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:21.822051 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume podName:06197912-3b3e-40aa-a19b-8025350bf804 nodeName:}" failed. No retries permitted until 2026-10-31 23:21:29.822035523 +0000 UTC m=+1916.051589912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume") pod "payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" (UID: "06197912-3b3e-40aa-a19b-8025350bf804") : configmap "payload-processing-plugins-e2e-mt-85b90bac" not found Oct 31 23:21:21.862342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.862316 5043 scope.go:122] "RemoveContainer" containerID="dd30efa8a12e0610cfc599f98ec5b9f0b7afd478359b89c0aacbb2e6e2ee7306" Oct 31 23:21:21.893652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.893620 5043 scope.go:122] "RemoveContainer" containerID="3c8cb29237b873b0eebc56508749bc30b4b774b2bbae682727533f504f0ecb9d" Oct 31 23:21:21.953889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:21.953862 5043 scope.go:122] "RemoveContainer" containerID="2db53418ce48d5ec216a92c23ddad4a0fce7ce90e7f78a29a260a4254e8f809d" Oct 31 23:21:22.027376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.027207 5043 generic.go:356] "Generic (PLEG): container finished" podID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerID="d92b844ec596fa77e27176e042eca461aad29ad582463d39277820c3fc5f9018" exitCode=0 Oct 31 23:21:22.027376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.027364 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:22.028513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.028481 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" event={"ID":"47d31628-1d5e-4601-9b3f-3b514fed875e","Type":"ContainerDied","Data":"d92b844ec596fa77e27176e042eca461aad29ad582463d39277820c3fc5f9018"} Oct 31 23:21:22.034497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.034468 5043 scope.go:122] "RemoveContainer" containerID="d3a9872839632cf1b19122316bee5f7211ad004afa8d3e9cb32ddd9034fa34a2" Oct 31 23:21:22.145496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.145473 5043 scope.go:122] "RemoveContainer" containerID="a13fea42703a17a3195408a68a6ea21298ebe23c4626ea8ba78a56f66d27787b" Oct 31 23:21:22.242903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.242883 5043 scope.go:122] "RemoveContainer" containerID="d170adf03be7977f19cbef8478742288e051fbd4379b2298094b536855160750" Oct 31 23:21:22.397232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.397208 5043 scope.go:122] "RemoveContainer" containerID="2fef461a9c8e9ead1ff06ed758f90dbdda6058511bda170ca4efed4e112edde8" Oct 31 23:21:22.425174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.425145 5043 scope.go:122] "RemoveContainer" containerID="aca3c88baebe1fd0c78139d02f31c89c325d6dc5c6b6774b09a686a9048aef74" Oct 31 23:21:22.578341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.578321 5043 scope.go:122] "RemoveContainer" containerID="35028b5c725596f0c7eb5316b6784342542a9b695b9c71bac261142a135a2c33" Oct 31 23:21:22.656457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.656434 5043 scope.go:122] "RemoveContainer" containerID="57393695f034569221bf98f7c5704425eef6c01f009378a81b74b10e657d3c20" Oct 31 23:21:22.693097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.693065 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:22.693182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.693105 5043 generic.go:356] "Generic (PLEG): container finished" podID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerID="c80e9eb07b05228715ce9d5fb8f1ea4fc46c76023c0c99f5b2867ca1725362a9" exitCode=0 Oct 31 23:21:22.693271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.693250 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" event={"ID":"f81afb95-ac07-4546-bea2-dfc33caf888d","Type":"ContainerDied","Data":"c80e9eb07b05228715ce9d5fb8f1ea4fc46c76023c0c99f5b2867ca1725362a9"} Oct 31 23:21:22.695526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.695502 5043 generic.go:356] "Generic (PLEG): container finished" podID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerID="37102e32d4a7f763663cdf7c0120f8d4a6ac2dd5f783fab9ffd21b1f5b79ae2e" exitCode=0 Oct 31 23:21:22.695703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.695609 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" event={"ID":"9a79e0dc-3d9b-4309-af30-92c2d76e8c41","Type":"ContainerDied","Data":"37102e32d4a7f763663cdf7c0120f8d4a6ac2dd5f783fab9ffd21b1f5b79ae2e"} Oct 31 23:21:22.698438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.698337 5043 generic.go:356] "Generic (PLEG): container finished" podID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerID="35469d41d88c962f7545fe8df512a334b4e1c24bd26d8c77412b355fac00cf50" exitCode=0 Oct 31 23:21:22.698438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.698416 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" event={"ID":"41cbfec0-981c-4be5-b984-0f1b8d71cc59","Type":"ContainerDied","Data":"35469d41d88c962f7545fe8df512a334b4e1c24bd26d8c77412b355fac00cf50"} Oct 31 23:21:22.703238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.703214 5043 generic.go:356] "Generic (PLEG): container finished" podID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerID="3d8fadd7f7cde659665f33db449490030d5933036832b9df1eab3dc50d79cd5e" exitCode=0 Oct 31 23:21:22.703355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.703285 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" event={"ID":"842a6154-7c20-43e2-bc07-03a6cce2d71b","Type":"ContainerDied","Data":"3d8fadd7f7cde659665f33db449490030d5933036832b9df1eab3dc50d79cd5e"} Oct 31 23:21:22.706032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.705768 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g"] Oct 31 23:21:22.706903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.706877 5043 generic.go:356] "Generic (PLEG): container finished" podID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerID="4854b34e74d727bf0de49827be4ab51521d669dc74f5a4b16eaee49d3b7f1376" exitCode=0 Oct 31 23:21:22.710401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.710372 5043 generic.go:356] "Generic (PLEG): container finished" podID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerID="0031786746efc4e173db89f8d083137c09cae82b1884156665e1f8e327d516ab" exitCode=0 Oct 31 23:21:22.713689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.713658 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:21:22.735637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.735611 5043 scope.go:122] "RemoveContainer" containerID="30638dc7a7bc664225976b15d39c9a7cf87dfbf38f6b467f96c16f00f4994847" Oct 31 23:21:23.001369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:22.928366 5043 scope.go:122] "RemoveContainer" containerID="92ce79b766b91216e54278ebc6a81fdbc9294e59c44595b801fb22e226e16d7a" Oct 31 23:21:23.144984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.108978 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:23.203338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.203250 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" podUID="b4fd03ee-af94-4df1-b834-eced676d664e" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.34:8081/readyz\": dial tcp 10.132.0.34:8081: connect: connection refused" Oct 31 23:21:23.206431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.206407 5043 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-vkldg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Oct 31 23:21:23.206616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.206441 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" podUID="12ea1ea8-83a9-4c99-a124-5bf66d1f26b5" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Oct 31 23:21:23.215263 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.215240 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" podUID="0063f042-8457-40ab-97d7-94d1420af56b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": dial tcp 10.132.0.33:8081: connect: connection refused" Oct 31 23:21:23.243782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.243744 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" event={"ID":"33eab6b9-e331-48da-bfbf-0d04e0baf215","Type":"ContainerDied","Data":"4854b34e74d727bf0de49827be4ab51521d669dc74f5a4b16eaee49d3b7f1376"} Oct 31 23:21:23.243941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.243809 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" event={"ID":"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba","Type":"ContainerDied","Data":"0031786746efc4e173db89f8d083137c09cae82b1884156665e1f8e327d516ab"} Oct 31 23:21:23.243941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.243835 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46"] Oct 31 23:21:23.243941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.243867 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.344933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.250030 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk45w\" (UniqueName: \"kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.344933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.250262 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.344933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.250320 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.352106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.352078 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk45w\" (UniqueName: \"kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.352322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.352306 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.352402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.352385 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.353100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.353065 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.354591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.354568 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.370410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.370387 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk45w\" (UniqueName: \"kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w\") pod \"maas-api-revoke-sub-255f7dcf-bl96g\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.450821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.450796 5043 scope.go:122] "RemoveContainer" containerID="ce1ef4ad441f0678c50bfc1ca65e872f59e589856843c0ec71fdced72b7a153c" Oct 31 23:21:23.639699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.575113 5043 patch_prober.go:29] interesting pod/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 23:21:23.639699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.575156 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 23:21:23.656013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.655984 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g"] Oct 31 23:21:23.656119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.656037 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46"] Oct 31 23:21:23.656119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.656080 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.711684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.711646 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" containerID="cri-o://6bf56397124ce158b9f46a647c9478377b71d2ee07f31193a58e705fc692976a" gracePeriod=25 Oct 31 23:21:23.712495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.712469 5043 scope.go:122] "RemoveContainer" containerID="591c4d54328c1ef5555703858cdf9b54e0cfd167c74b6b1e4f723fa72529442a" Oct 31 23:21:23.759141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.759099 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.759343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.759328 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.759462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.759444 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcktj\" (UniqueName: \"kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.779422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.779190 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:23.849871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.849846 5043 scope.go:122] "RemoveContainer" containerID="357e746c74a73eece1a6bd4bfdb496afcb437434eb140cb40b92d28ff0d2d8de" Oct 31 23:21:23.860749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.860716 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.860971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.860952 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.861084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.861068 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcktj\" (UniqueName: \"kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.861643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.861621 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.863174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.863152 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:23.877690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:23.877659 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcktj\" (UniqueName: \"kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj\") pod \"maas-api-revoke-sub-a1512cc4-skk46\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:24.034584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.034158 5043 patch_prober.go:29] interesting pod/jobset-controller-manager-64f549b4b7-q5w9p container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused" start-of-body= Oct 31 23:21:24.034584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.034212 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" podUID="863f9cb3-dbae-4b56-b240-eb661de1f2dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.41:8081/readyz\": dial tcp 10.132.0.41:8081: connect: connection refused" Oct 31 23:21:24.317911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.317879 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:24.318060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.317993 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" event={"ID":"863f9cb3-dbae-4b56-b240-eb661de1f2dc","Type":"ContainerStarted","Data":"b11ff28e4a2933d977b3bde5e8a159d5841c294464fc05337429b05a49e1b1b0"} Oct 31 23:21:24.318170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.318152 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 23:21:24.327133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.327028 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" event={"ID":"b4fd03ee-af94-4df1-b834-eced676d664e","Type":"ContainerStarted","Data":"9678d958051e5a7b1fa91e35dfc23f6aeedf381ed88044ae9bfcf286ffba675f"} Oct 31 23:21:24.327133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.327092 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 23:21:24.332954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.332931 5043 generic.go:356] "Generic (PLEG): container finished" podID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerID="a112283e701bccf3ade34448bb910c025ecbd7961915ee71787f35773df4709c" exitCode=0 Oct 31 23:21:24.333218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.333093 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" event={"ID":"b6a2e7de-778e-47c1-a35a-d4f066df6748","Type":"ContainerDied","Data":"a112283e701bccf3ade34448bb910c025ecbd7961915ee71787f35773df4709c"} Oct 31 23:21:24.337758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.337732 5043 generic.go:356] "Generic (PLEG): container finished" podID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerID="1e3c5dd51bbb61614ccbdce07be2fbcbd5083fc2914e57121c3ec8733122feac" exitCode=0 Oct 31 23:21:24.337835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.337800 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" event={"ID":"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f","Type":"ContainerDied","Data":"1e3c5dd51bbb61614ccbdce07be2fbcbd5083fc2914e57121c3ec8733122feac"} Oct 31 23:21:24.337893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.337844 5043 scope.go:122] "RemoveContainer" containerID="9bc61faa5e3eb2579d4381e335c0430ff7e623de98b614b5d894a0730177223e" Oct 31 23:21:24.340913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.340895 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" event={"ID":"485a8e4b-a74c-4463-affc-44a6374fb81c","Type":"ContainerStarted","Data":"c713142661e13ad0c6c971fa11a93b2d5cb2e6af455b6911b55929c9ddf8d601"} Oct 31 23:21:24.341031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.341016 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 23:21:24.342995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.342978 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" event={"ID":"e0a050c1-60b2-4ee9-a860-8f646427645e","Type":"ContainerStarted","Data":"26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c"} Oct 31 23:21:24.343081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.343027 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:21:24.345000 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.344979 5043 generic.go:356] "Generic (PLEG): container finished" podID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerID="a0adc13a333dfdbba56b1a5e29430c78284ab010e8f20e3ad316f69d16f26947" exitCode=0 Oct 31 23:21:24.345097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.345050 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" event={"ID":"1327691f-6ea3-43b8-8b4b-9e38673bbb03","Type":"ContainerDied","Data":"a0adc13a333dfdbba56b1a5e29430c78284ab010e8f20e3ad316f69d16f26947"} Oct 31 23:21:24.349383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.349355 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:21:24.349493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.349472 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" event={"ID":"12ea1ea8-83a9-4c99-a124-5bf66d1f26b5","Type":"ContainerStarted","Data":"dca0c7692d1c42ef9df254dd9a880ee837614c5aa4db2c2fcea79552fb5cccd1"} Oct 31 23:21:24.349583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.349521 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 23:21:24.353879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.353861 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" event={"ID":"cdefa176-e223-4de2-857d-14d6ffecf8a7","Type":"ContainerStarted","Data":"5c3fd2956c3fd005d081ebfa51ff942bea13eddd665af00afbb94cbbfdf68e1c"} Oct 31 23:21:24.354598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.354580 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 23:21:24.439801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.354779 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-vkldg" Oct 31 23:21:24.439801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.357853 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" event={"ID":"677d053b-fbf4-4b76-9782-6b3235dc340f","Type":"ContainerStarted","Data":"9aefb8fe9fef45ea887006332cab41255b795fe96f0210a2e937e9b0d9da9586"} Oct 31 23:21:24.439801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.358165 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 23:21:24.439801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.360522 5043 generic.go:356] "Generic (PLEG): container finished" podID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerID="e29ef966fe50e855687948706d2a78d30d1f4e59383978322d4d6003ad55b408" exitCode=0 Oct 31 23:21:24.439801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.360595 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" event={"ID":"651744fd-77a3-431f-8f0e-a2f8509203f5","Type":"ContainerDied","Data":"e29ef966fe50e855687948706d2a78d30d1f4e59383978322d4d6003ad55b408"} Oct 31 23:21:24.441426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.441253 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:24.489197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.489169 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 23:21:24.489333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.489216 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 23:21:24.707895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.707853 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:24.876085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.876053 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 23:21:24.876269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:24.876108 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.127005 5043 patch_prober.go:29] interesting pod/router-default-8f9bc98f7-xn5rp container/router namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]backend-proxy-http ok Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: [+]has-synced ok Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: [-]process-running failed: reason withheld Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: healthz check failed Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.127052 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" podUID="7afda5df-8528-4304-aa1d-156aefde0156" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.363626 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.363737 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.400498 5043 generic.go:356] "Generic (PLEG): container finished" podID="06197912-3b3e-40aa-a19b-8025350bf804" containerID="6bf56397124ce158b9f46a647c9478377b71d2ee07f31193a58e705fc692976a" exitCode=0 Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.400580 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" event={"ID":"06197912-3b3e-40aa-a19b-8025350bf804","Type":"ContainerDied","Data":"6bf56397124ce158b9f46a647c9478377b71d2ee07f31193a58e705fc692976a"} Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.402891 5043 generic.go:356] "Generic (PLEG): container finished" podID="51479f0b-8c1b-472e-b4aa-80a088062549" containerID="b338d82c0b906c30d6ac55ae97fc8626e6df193597e361a5bfe39359cf9c4f11" exitCode=0 Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.403657 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" event={"ID":"51479f0b-8c1b-472e-b4aa-80a088062549","Type":"ContainerDied","Data":"b338d82c0b906c30d6ac55ae97fc8626e6df193597e361a5bfe39359cf9c4f11"} Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:25.583797 5043 logging.go:55] [core] [Channel #934 SubChannel #935]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.184:50052", ServerName: "10.132.0.184:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.184:50052: connect: connection refused" Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.756313 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.805277 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc"] Oct 31 23:21:26.145692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:25.805685 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" containerID="cri-o://ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1" gracePeriod=30 Oct 31 23:21:26.406979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.320658 5043 patch_prober.go:29] interesting pod/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 31 23:21:26.406979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.320724 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 31 23:21:26.406979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.352834 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:26.424258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.423768 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:21:26.424258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.423840 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:21:26.487058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.487027 5043 patch_prober.go:29] interesting pod/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 23:21:26.487205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.487074 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 23:21:26.549616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.549587 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:21:26.549794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.549642 5043 generic.go:356] "Generic (PLEG): container finished" podID="7afda5df-8528-4304-aa1d-156aefde0156" containerID="5106193ed3e2ad49ca55a53cda62c6625603ac001a10b2f0d38402b7c1ee3049" exitCode=137 Oct 31 23:21:26.549794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.549731 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" event={"ID":"7afda5df-8528-4304-aa1d-156aefde0156","Type":"ContainerDied","Data":"5106193ed3e2ad49ca55a53cda62c6625603ac001a10b2f0d38402b7c1ee3049"} Oct 31 23:21:26.582162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.582109 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.184:50052\" within 1s: context deadline exceeded" Oct 31 23:21:26.582587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.582528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:26.692539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.692512 5043 scope.go:122] "RemoveContainer" containerID="4af9753839e3c7534378a9d6f829bcf5f99911cb6daed1b3b97afbdbd759327f" Oct 31 23:21:26.944419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:26.944332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:27.310626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:27.310468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:27.750582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:27.743780 5043 scope.go:122] "RemoveContainer" containerID="536d984fedc32cc840d13b1219849d009cb536d3fa27ebff21b72dd1785df755" Oct 31 23:21:28.423755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.423645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:28.548066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.547897 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:21:28.592179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.592151 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:21:28.599195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.598146 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" event={"ID":"06197912-3b3e-40aa-a19b-8025350bf804","Type":"ContainerDied","Data":"8802eea9293b8abe3d56ca03197700a8026ef5a64403360c0bb4303d9085d089"} Oct 31 23:21:28.599195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.598232 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8802eea9293b8abe3d56ca03197700a8026ef5a64403360c0bb4303d9085d089" Oct 31 23:21:28.601363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.601335 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" event={"ID":"47d31628-1d5e-4601-9b3f-3b514fed875e","Type":"ContainerDied","Data":"e6232f4ee35c096c18f3e24bc8dcaf893c3166b6cf3e0615c2e90c410abeba1d"} Oct 31 23:21:28.601494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.601385 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6232f4ee35c096c18f3e24bc8dcaf893c3166b6cf3e0615c2e90c410abeba1d" Oct 31 23:21:28.604560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.604519 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" event={"ID":"651744fd-77a3-431f-8f0e-a2f8509203f5","Type":"ContainerDied","Data":"138c4da78ba86962a0e250d129360ec56e257547965e459f5907a4cc233f67f4"} Oct 31 23:21:28.604664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.604580 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="138c4da78ba86962a0e250d129360ec56e257547965e459f5907a4cc233f67f4" Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.606858 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.606937 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.606980 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607023 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607058 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607117 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607133 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607177 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607226 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607252 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607276 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607298 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607336 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607382 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607458 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607476 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs\") pod \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\" (UID: \"9a79e0dc-3d9b-4309-af30-92c2d76e8c41\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607726 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.608704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.607803 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs\") pod \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\" (UID: \"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba\") " Oct 31 23:21:28.609482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.609459 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.610580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.610343 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.610580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.610472 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.611044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.611020 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.611428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.611081 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.611428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.611257 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.611428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.611293 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.611910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.611730 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.611987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.611945 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.612061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.612001 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.635968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.635937 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:21:28.691955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.662784 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:21:28.691955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.676775 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" event={"ID":"0063f042-8457-40ab-97d7-94d1420af56b","Type":"ContainerStarted","Data":"170937d003e487b563d868e9a34d5ee271bd9267bda00a45ad37f89ddae391d7"} Oct 31 23:21:28.691955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.676842 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 23:21:28.691955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.679164 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" event={"ID":"f81afb95-ac07-4546-bea2-dfc33caf888d","Type":"ContainerDied","Data":"a4a8a4432216e13a6777d53b99bcd8759fdcdf333abe994ae8c94f5fa1a8b875"} Oct 31 23:21:28.691955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.679204 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a8a4432216e13a6777d53b99bcd8759fdcdf333abe994ae8c94f5fa1a8b875" Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.713400 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.714166 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.714240 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.714313 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.714387 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.714438 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.714524 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.715058 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.715611 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd\" (UniqueName: \"kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd\") pod \"06197912-3b3e-40aa-a19b-8025350bf804\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.715700 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.715778 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket\") pod \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\" (UID: \"1327691f-6ea3-43b8-8b4b-9e38673bbb03\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.715892 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume\") pod \"06197912-3b3e-40aa-a19b-8025350bf804\" (UID: \"06197912-3b3e-40aa-a19b-8025350bf804\") " Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.718509 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.718534 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.721009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.719970 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.722763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.722607 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.725571 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" event={"ID":"51479f0b-8c1b-472e-b4aa-80a088062549","Type":"ContainerDied","Data":"73130c78485bb8cde334b5b8c1d5c98e18bb5800a00d514890e60bfcf9d92b72"} Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.725629 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73130c78485bb8cde334b5b8c1d5c98e18bb5800a00d514890e60bfcf9d92b72" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.726764 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:28.726824 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95df7923_e428_495b_a3a4_ea99cb1b013e.slice/crio-517d2682702a21e4caf1cff07ec2f18830a4204c0ba15568137143b543e6f5cd WatchSource:0}: Error finding container 517d2682702a21e4caf1cff07ec2f18830a4204c0ba15568137143b543e6f5cd: Status 404 returned error can't find the container with id 517d2682702a21e4caf1cff07ec2f18830a4204c0ba15568137143b543e6f5cd Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.731468 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.731573 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732353 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732380 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732410 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732425 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732439 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732454 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732468 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732485 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732498 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.732911 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume" pod "06197912-3b3e-40aa-a19b-8025350bf804" (UID: "06197912-3b3e-40aa-a19b-8025350bf804"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.733272 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "kube-api-access-zkzv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.733962 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.734056 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.747696 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw" pod "9a79e0dc-3d9b-4309-af30-92c2d76e8c41" (UID: "9a79e0dc-3d9b-4309-af30-92c2d76e8c41"). InnerVolumeSpecName "kube-api-access-85gsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.747761 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.748804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.747847 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.767114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.752687 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd" pod "06197912-3b3e-40aa-a19b-8025350bf804" (UID: "06197912-3b3e-40aa-a19b-8025350bf804"). InnerVolumeSpecName "kube-api-access-l28hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.767114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.758123 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy" pod "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" (UID: "be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.767114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.764241 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:28.767114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.764308 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:28.767114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.764348 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:28.774903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.774879 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46"] Oct 31 23:21:28.778586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.778516 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" event={"ID":"9a79e0dc-3d9b-4309-af30-92c2d76e8c41","Type":"ContainerDied","Data":"e8522cba4052bf65eca0e4b240e41da55de974c165aece0b14844b7f3320b034"} Oct 31 23:21:28.778722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.778598 5043 scope.go:122] "RemoveContainer" containerID="37102e32d4a7f763663cdf7c0120f8d4a6ac2dd5f783fab9ffd21b1f5b79ae2e" Oct 31 23:21:28.778784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.778760 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx" Oct 31 23:21:28.782120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.782016 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct"] Oct 31 23:21:28.788399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.788373 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.791055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.790954 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf"] Oct 31 23:21:28.792261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.792234 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.792375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.792319 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "kube-api-access-jv78v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.792375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.792352 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token" pod "1327691f-6ea3-43b8-8b4b-9e38673bbb03" (UID: "1327691f-6ea3-43b8-8b4b-9e38673bbb03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.797013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.796143 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" event={"ID":"41cbfec0-981c-4be5-b984-0f1b8d71cc59","Type":"ContainerDied","Data":"944d3bdda09e6dfc66cdd2cbea209d11c2b233d9b5c6960a14e95370855da153"} Oct 31 23:21:28.797013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.796176 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944d3bdda09e6dfc66cdd2cbea209d11c2b233d9b5c6960a14e95370855da153" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.812197 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj"] Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.813921 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.828208 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833178 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833268 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833370 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833416 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833450 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833529 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833630 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833826 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833881 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data\") pod \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\" (UID: \"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f\") " Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.833928 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834839 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834864 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834880 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv78v\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-kube-api-access-jv78v\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834895 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834911 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834925 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834940 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834955 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834971 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkzv8\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-kube-api-access-zkzv8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.834986 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835000 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l28hd\" (UniqueName: \"kubernetes.io/projected/06197912-3b3e-40aa-a19b-8025350bf804-kube-api-access-l28hd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835015 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835029 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835042 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06197912-3b3e-40aa-a19b-8025350bf804-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835057 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85gsw\" (UniqueName: \"kubernetes.io/projected/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-kube-api-access-85gsw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835072 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a79e0dc-3d9b-4309-af30-92c2d76e8c41-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835087 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835100 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1327691f-6ea3-43b8-8b4b-9e38673bbb03-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835113 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.836352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.835608 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.836488 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.837304 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.837646 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" event={"ID":"842a6154-7c20-43e2-bc07-03a6cce2d71b","Type":"ContainerDied","Data":"c6b6bb3836845ad3f4da9ac051230adc0f246617d8f784fc7c6e92100bbd2ece"} Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.837694 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b6bb3836845ad3f4da9ac051230adc0f246617d8f784fc7c6e92100bbd2ece" Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.838139 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.838378 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.839184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.838591 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.842103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.839865 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.842103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.841211 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "kube-api-access-pflvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.842103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.841263 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:21:28.842103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.841437 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" event={"ID":"7afda5df-8528-4304-aa1d-156aefde0156","Type":"ContainerStarted","Data":"8d67b1b72e2dd8b31dd99ad94442ed4b362d2a72f2c7a3221f9b899f0bc81a59"} Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.844774 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo" pod "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" (UID: "f9bb011d-a74e-4d78-8a04-79f7eb1dff8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.847922 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.841989 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.851811 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" event={"ID":"b6a2e7de-778e-47c1-a35a-d4f066df6748","Type":"ContainerDied","Data":"0467786627e3156d624c9a0bc2db4a47fe5787e3e810e9d043ca61837d0aca54"} Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.851847 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0467786627e3156d624c9a0bc2db4a47fe5787e3e810e9d043ca61837d0aca54" Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.851898 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 23:21:28.852172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.851918 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6" event={"ID":"be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba","Type":"ContainerDied","Data":"dc9c4d3d15ec3238b22dac69d0450d456b4adab4a550e6c37de2890ad68195dc"} Oct 31 23:21:28.862514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.854769 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:21:28.862514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.857257 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" event={"ID":"f9bb011d-a74e-4d78-8a04-79f7eb1dff8f","Type":"ContainerDied","Data":"5e79ec67ea01f6921b29da87cab3e0fc5bfca294d2c51fecdbb9755cefe249a6"} Oct 31 23:21:28.862514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.857356 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl" Oct 31 23:21:28.862514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.861004 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" event={"ID":"1327691f-6ea3-43b8-8b4b-9e38673bbb03","Type":"ContainerDied","Data":"cdaeac02295a1607313310f89b938aea56a0dbb463ef04692ac587370f23ac5e"} Oct 31 23:21:28.862514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.861085 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls" Oct 31 23:21:28.892154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.892125 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:21:28.913232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.913209 5043 scope.go:122] "RemoveContainer" containerID="0031786746efc4e173db89f8d083137c09cae82b1884156665e1f8e327d516ab" Oct 31 23:21:28.913912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.913890 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:21:28.935908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.935874 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.936097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.935927 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.936097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.935962 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.936097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.935992 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.936097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936034 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.936097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936064 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936099 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936131 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936177 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936215 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936243 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936302 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936343 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936406 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936474 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936501 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936534 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936575 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936609 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936654 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936763 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.936827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936824 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936885 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936943 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.936972 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937012 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937075 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937121 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937178 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.937270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937251 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937281 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937323 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937389 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937447 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937517 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937615 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937650 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937690 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo\") pod \"842a6154-7c20-43e2-bc07-03a6cce2d71b\" (UID: \"842a6154-7c20-43e2-bc07-03a6cce2d71b\") " Oct 31 23:21:28.937754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937728 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.938195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937776 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.938195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937822 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo\") pod \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\" (UID: \"41cbfec0-981c-4be5-b984-0f1b8d71cc59\") " Oct 31 23:21:28.938195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937885 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo\") pod \"47d31628-1d5e-4601-9b3f-3b514fed875e\" (UID: \"47d31628-1d5e-4601-9b3f-3b514fed875e\") " Oct 31 23:21:28.938195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937954 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.938195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.937988 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token\") pod \"651744fd-77a3-431f-8f0e-a2f8509203f5\" (UID: \"651744fd-77a3-431f-8f0e-a2f8509203f5\") " Oct 31 23:21:28.938195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.938038 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo\") pod \"51479f0b-8c1b-472e-b4aa-80a088062549\" (UID: \"51479f0b-8c1b-472e-b4aa-80a088062549\") " Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939402 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939431 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939446 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflvl\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-kube-api-access-pflvl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939463 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939477 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939491 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939506 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.940371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.939520 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:28.941145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.941034 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.941262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.941177 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.941578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.941405 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.942882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.941718 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.942882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.941927 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.942882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.942259 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.942882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.942275 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.942882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.942590 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.942882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.942855 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.944325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.944045 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.945652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.945443 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.955669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.955629 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.956019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.955903 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.956019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.955972 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.956183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.956054 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:21:28.957342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.956287 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.957342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.956422 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "kube-api-access-7llpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.957342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.956499 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "kube-api-access-zl9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.957342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.957106 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.957342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.957266 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "kube-api-access-z9gvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.957342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.957314 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.958206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.958187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.958324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.958221 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.958432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.958409 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.958514 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.958220 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.958611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.958237 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.958907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.958885 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "kube-api-access-lh68q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.959275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.959254 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "kube-api-access-cw89x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.960821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.960589 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.960821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.960782 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.960821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.960795 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.961103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.961005 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:28.961398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.961370 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.969865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.969837 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.969865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.969854 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo" pod "47d31628-1d5e-4601-9b3f-3b514fed875e" (UID: "47d31628-1d5e-4601-9b3f-3b514fed875e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.983242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.983193 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.983398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.983313 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.983398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.983366 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.989621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.989591 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.989722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.989633 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.989722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.989669 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token" pod "651744fd-77a3-431f-8f0e-a2f8509203f5" (UID: "651744fd-77a3-431f-8f0e-a2f8509203f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:28.992473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.992107 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.992473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.992173 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data" pod "842a6154-7c20-43e2-bc07-03a6cce2d71b" (UID: "842a6154-7c20-43e2-bc07-03a6cce2d71b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.992473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.992196 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:21:28.992473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.992378 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy" pod "51479f0b-8c1b-472e-b4aa-80a088062549" (UID: "51479f0b-8c1b-472e-b4aa-80a088062549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:28.992473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.992394 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:28.992473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.992449 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data" pod "41cbfec0-981c-4be5-b984-0f1b8d71cc59" (UID: "41cbfec0-981c-4be5-b984-0f1b8d71cc59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.004823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:28.998975 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 23:21:29.027296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.026472 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041249 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041328 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041357 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041432 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041475 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041505 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041534 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041623 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041698 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041757 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041788 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041905 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041944 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.041997 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.042129 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.042193 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.042293 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data\") pod \"b6a2e7de-778e-47c1-a35a-d4f066df6748\" (UID: \"b6a2e7de-778e-47c1-a35a-d4f066df6748\") " Oct 31 23:21:29.044016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.042343 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert\") pod \"f81afb95-ac07-4546-bea2-dfc33caf888d\" (UID: \"f81afb95-ac07-4546-bea2-dfc33caf888d\") " Oct 31 23:21:29.046324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046299 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046330 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046345 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046358 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046371 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51479f0b-8c1b-472e-b4aa-80a088062549-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046386 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw89x\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-kube-api-access-cw89x\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046400 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046414 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046427 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llpk\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-kube-api-access-7llpk\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046443 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046457 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046471 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl9pd\" (UniqueName: \"kubernetes.io/projected/651744fd-77a3-431f-8f0e-a2f8509203f5-kube-api-access-zl9pd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046486 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046502 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046516 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046531 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046597 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046613 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046631 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh68q\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-kube-api-access-lh68q\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046644 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046665 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046682 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046696 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046709 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046723 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9gvd\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-kube-api-access-z9gvd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046739 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d31628-1d5e-4601-9b3f-3b514fed875e-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046752 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046768 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d31628-1d5e-4601-9b3f-3b514fed875e-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046782 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/842a6154-7c20-43e2-bc07-03a6cce2d71b-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046796 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cbfec0-981c-4be5-b984-0f1b8d71cc59-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046810 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046825 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046839 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51479f0b-8c1b-472e-b4aa-80a088062549-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046853 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046867 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651744fd-77a3-431f-8f0e-a2f8509203f5-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046881 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/842a6154-7c20-43e2-bc07-03a6cce2d71b-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046896 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651744fd-77a3-431f-8f0e-a2f8509203f5-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.046900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046910 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51479f0b-8c1b-472e-b4aa-80a088062549-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046923 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046938 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046952 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cbfec0-981c-4be5-b984-0f1b8d71cc59-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046967 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651744fd-77a3-431f-8f0e-a2f8509203f5-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046981 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/842a6154-7c20-43e2-bc07-03a6cce2d71b-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.046996 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d31628-1d5e-4601-9b3f-3b514fed875e-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.047009 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51479f0b-8c1b-472e-b4aa-80a088062549-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.047327 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:29.068896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.047799 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.083297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.081444 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.083297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.081570 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.083871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.083766 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "kube-api-access-wh8mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:29.101386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.088769 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.101386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.089051 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:29.101386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.090852 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.101386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.093829 5043 scope.go:122] "RemoveContainer" containerID="1e3c5dd51bbb61614ccbdce07be2fbcbd5083fc2914e57121c3ec8733122feac" Oct 31 23:21:29.104588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.101605 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:29.117297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.114330 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls"] Oct 31 23:21:29.118733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.118500 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token" pod "b6a2e7de-778e-47c1-a35a-d4f066df6748" (UID: "b6a2e7de-778e-47c1-a35a-d4f066df6748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:29.126848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.123713 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.126848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.123857 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.126848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.123877 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.126848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.124445 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.139575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.133744 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-vj9ls"] Oct 31 23:21:29.147533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.147508 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.147702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.147672 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.147902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.147884 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148011 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148111 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148192 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148226 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148265 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148297 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data\") pod \"33eab6b9-e331-48da-bfbf-0d04e0baf215\" (UID: \"33eab6b9-e331-48da-bfbf-0d04e0baf215\") " Oct 31 23:21:29.148968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148950 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148972 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.148987 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a2e7de-778e-47c1-a35a-d4f066df6748-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149000 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149013 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149027 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149040 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149053 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149066 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8mz\" (UniqueName: \"kubernetes.io/projected/b6a2e7de-778e-47c1-a35a-d4f066df6748-kube-api-access-wh8mz\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.149075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149079 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.150145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149093 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.150145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149108 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.150145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149122 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a2e7de-778e-47c1-a35a-d4f066df6748-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.150145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149136 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81afb95-ac07-4546-bea2-dfc33caf888d-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.150145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.149956 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:29.199431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.158513 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.199431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.164101 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.199431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.164092 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.199431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.199299 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.203324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.203248 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d"] Oct 31 23:21:29.225236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.224813 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.225236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.224839 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "kube-api-access-l74cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:29.225236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.224816 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:29.225236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.224883 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "kube-api-access-5j9cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:29.225236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.225127 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token" pod "f81afb95-ac07-4546-bea2-dfc33caf888d" (UID: "f81afb95-ac07-4546-bea2-dfc33caf888d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:29.225236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.225177 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:29.234016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.233309 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:29.234404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.234369 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy" pod "33eab6b9-e331-48da-bfbf-0d04e0baf215" (UID: "33eab6b9-e331-48da-bfbf-0d04e0baf215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:29.240953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.240928 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr"] Oct 31 23:21:29.241716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.241612 5043 scope.go:122] "RemoveContainer" containerID="a0adc13a333dfdbba56b1a5e29430c78284ab010e8f20e3ad316f69d16f26947" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251395 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251425 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251441 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251454 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251467 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251482 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74cc\" (UniqueName: \"kubernetes.io/projected/33eab6b9-e331-48da-bfbf-0d04e0baf215-kube-api-access-l74cc\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251496 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251511 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251526 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251540 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33eab6b9-e331-48da-bfbf-0d04e0baf215-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251600 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33eab6b9-e331-48da-bfbf-0d04e0baf215-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251614 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j9cc\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-kube-api-access-5j9cc\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.251657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.251630 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81afb95-ac07-4546-bea2-dfc33caf888d-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:29.320601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.320517 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6"] Oct 31 23:21:29.348197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.348168 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l"] Oct 31 23:21:29.358163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.358140 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s"] Oct 31 23:21:29.365006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.364982 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-gz7m6"] Oct 31 23:21:29.370557 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.370527 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79"] Oct 31 23:21:29.389724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.389701 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g"] Oct 31 23:21:29.443413 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:29.439152 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795bdc7a_da31_4a12_afad_cdc0ad0f7fed.slice/crio-8e199f978f4ac490a496a8ea8ab0e36b6b1b4b9bc6532a3fbd08312960dcf842 WatchSource:0}: Error finding container 8e199f978f4ac490a496a8ea8ab0e36b6b1b4b9bc6532a3fbd08312960dcf842: Status 404 returned error can't find the container with id 8e199f978f4ac490a496a8ea8ab0e36b6b1b4b9bc6532a3fbd08312960dcf842 Oct 31 23:21:29.521800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.521759 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl"] Oct 31 23:21:29.532352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.532326 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-x2vbl"] Oct 31 23:21:29.863413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.863234 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx"] Oct 31 23:21:29.890803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.890770 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" event={"ID":"c769ec16-9843-4e45-9caf-298744983e65","Type":"ContainerStarted","Data":"78852d1ffea1e452bca13c755e516b7def266dd3b199c930c73723528e76076c"} Oct 31 23:21:29.891013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.890831 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" event={"ID":"c769ec16-9843-4e45-9caf-298744983e65","Type":"ContainerStarted","Data":"9189f1dc092962f11a0efe3ab9c6571ea2abd46dba93685f978cd4dd39432ec2"} Oct 31 23:21:29.895400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.895367 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" event={"ID":"f4a81084-193c-40f0-abda-964c909230c0","Type":"ContainerStarted","Data":"c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6"} Oct 31 23:21:29.895503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.895426 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" event={"ID":"f4a81084-193c-40f0-abda-964c909230c0","Type":"ContainerStarted","Data":"bca25f4a4b8b2262458cced02e2625eeb2502175bb9b3869f17981153593e650"} Oct 31 23:21:29.895503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.895459 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:29.897353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.897321 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" event={"ID":"28549d8a-85b5-4060-b623-be279fa8d14f","Type":"ContainerStarted","Data":"524b01ff1d807fdc918747afe744812d05a5c26f2dcbe294165bed49d2fb89df"} Oct 31 23:21:29.899199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.899170 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" event={"ID":"bd32bc24-e23a-4a73-a804-00bb90c9c3e4","Type":"ContainerStarted","Data":"4ef836bcf9ad0a3fd64f15e3f59d3f64ed01347b2298312836cc55fe969566ef"} Oct 31 23:21:29.900636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.900604 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-tdknx"] Oct 31 23:21:29.901430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.901402 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" event={"ID":"15bd6f6f-3f7e-46d2-87e1-2d4437233318","Type":"ContainerStarted","Data":"b721301d75fe919f6f08fe8cc88f51f3a9e7e429ee0d73f98c954f61c08a5d94"} Oct 31 23:21:29.903560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.903526 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" event={"ID":"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa","Type":"ContainerStarted","Data":"7ed4a4e1449fb7f4eb8ac799443f4586dae7b28585f08c351625da6d17c41658"} Oct 31 23:21:29.905508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.905481 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" event={"ID":"95df7923-e428-495b-a3a4-ea99cb1b013e","Type":"ContainerStarted","Data":"5c73ad771236e3c3375ef292a802d5c1364e3dd14277c0f5525afe2a22c6c991"} Oct 31 23:21:29.905617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.905531 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" event={"ID":"95df7923-e428-495b-a3a4-ea99cb1b013e","Type":"ContainerStarted","Data":"517d2682702a21e4caf1cff07ec2f18830a4204c0ba15568137143b543e6f5cd"} Oct 31 23:21:29.908616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.908593 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" event={"ID":"7f3337f8-211c-4566-a0bd-521baacecd83","Type":"ContainerStarted","Data":"9524bc4e8d318a5d1636a198a0e027d0980430f74baecf2ef55920b1ac1088b0"} Oct 31 23:21:29.910001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.909975 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" event={"ID":"795bdc7a-da31-4a12-afad-cdc0ad0f7fed","Type":"ContainerStarted","Data":"8e199f978f4ac490a496a8ea8ab0e36b6b1b4b9bc6532a3fbd08312960dcf842"} Oct 31 23:21:29.912079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.912058 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" event={"ID":"33eab6b9-e331-48da-bfbf-0d04e0baf215","Type":"ContainerDied","Data":"7ae729751b5945f459e252110f10de415a2bb7a3bc19080f4fcabc4b3feaa909"} Oct 31 23:21:29.912156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.912100 5043 scope.go:122] "RemoveContainer" containerID="4854b34e74d727bf0de49827be4ab51521d669dc74f5a4b16eaee49d3b7f1376" Oct 31 23:21:29.912212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.912201 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb" Oct 31 23:21:29.914986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.914942 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" event={"ID":"51bcca4d-52c3-4681-a8c9-f7e090ae567a","Type":"ContainerStarted","Data":"25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43"} Oct 31 23:21:29.915090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.914994 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" event={"ID":"51bcca4d-52c3-4681-a8c9-f7e090ae567a","Type":"ContainerStarted","Data":"0b65c63b27e1b6e4799c1f27095fcccf1c94256f2dc2abe7be4e638bfe2e6346"} Oct 31 23:21:29.915864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.915842 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:29.919613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.917365 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd" Oct 31 23:21:29.919719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.919685 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm" Oct 31 23:21:29.919830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.919693 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4" Oct 31 23:21:29.919972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.919958 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq" Oct 31 23:21:29.920049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.919976 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp" Oct 31 23:21:29.928620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.921253 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87" Oct 31 23:21:29.945056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.938821 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l" Oct 31 23:21:29.945056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.940680 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml" Oct 31 23:21:29.956935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.954232 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" path="/var/lib/kubelet/pods/1327691f-6ea3-43b8-8b4b-9e38673bbb03/volumes" Oct 31 23:21:29.956935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.954978 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" path="/var/lib/kubelet/pods/9a79e0dc-3d9b-4309-af30-92c2d76e8c41/volumes" Oct 31 23:21:29.956935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.955692 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" path="/var/lib/kubelet/pods/be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba/volumes" Oct 31 23:21:29.956935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:29.956664 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" path="/var/lib/kubelet/pods/f9bb011d-a74e-4d78-8a04-79f7eb1dff8f/volumes" Oct 31 23:21:30.011113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.008598 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8f9bc98f7-xn5rp" Oct 31 23:21:30.058133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.058105 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:30.131593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.128777 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" podStartSLOduration=16.128759991 podStartE2EDuration="16.128759991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:30.126230942 +0000 UTC m=+1916.355785370" watchObservedRunningTime="2026-10-31 23:21:30.128759991 +0000 UTC m=+1916.358314403" Oct 31 23:21:30.172427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.172396 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp"] Oct 31 23:21:30.183041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.183000 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-6mhpp"] Oct 31 23:21:30.202396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.201246 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:30.248473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.206853 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:21:30.248473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.206951 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:21:30.248473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.211291 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4"] Oct 31 23:21:30.248473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.240313 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-v2kt4"] Oct 31 23:21:30.281422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.281382 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq"] Oct 31 23:21:30.291778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.291745 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-x9qtq"] Oct 31 23:21:30.293121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.293091 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:30.351456 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.351423 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm"] Oct 31 23:21:30.356960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.356932 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-85b90bac-6cdf6b9d4-j2vxm"] Oct 31 23:21:30.370406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.370376 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb"] Oct 31 23:21:30.377867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.377837 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-fqzdb"] Oct 31 23:21:30.406456 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.406426 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qpdv4" Oct 31 23:21:30.409404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.409378 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd"] Oct 31 23:21:30.424896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.424869 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-7s6bd"] Oct 31 23:21:30.441386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.441348 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml"] Oct 31 23:21:30.457259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.457228 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-85b90bac-openshift-default-6bcc4998bf-8f2ml"] Oct 31 23:21:30.489804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.489736 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podStartSLOduration=14.489718443 podStartE2EDuration="14.489718443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:30.48679047 +0000 UTC m=+1916.716344883" watchObservedRunningTime="2026-10-31 23:21:30.489718443 +0000 UTC m=+1916.719272853" Oct 31 23:21:30.556360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.556273 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87"] Oct 31 23:21:30.575352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.575320 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-9tf87"] Oct 31 23:21:30.589209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.589151 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" podStartSLOduration=16.589132202000002 podStartE2EDuration="16.589132202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:30.587084477 +0000 UTC m=+1916.816638890" watchObservedRunningTime="2026-10-31 23:21:30.589132202 +0000 UTC m=+1916.818686613" Oct 31 23:21:30.626168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.626112 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" podStartSLOduration=8.626094808 podStartE2EDuration="8.626094808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:30.622367796 +0000 UTC m=+1916.851922208" watchObservedRunningTime="2026-10-31 23:21:30.626094808 +0000 UTC m=+1916.855649212" Oct 31 23:21:30.647327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.647292 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l"] Oct 31 23:21:30.652347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.652315 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vfr6l"] Oct 31 23:21:30.932750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.932656 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" event={"ID":"28549d8a-85b5-4060-b623-be279fa8d14f","Type":"ContainerStarted","Data":"2758fbd6565169c3d9119226ad06d6bb67c0bf20bccb547ef5cc994e1cae9c60"} Oct 31 23:21:30.935203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.935176 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" event={"ID":"bd32bc24-e23a-4a73-a804-00bb90c9c3e4","Type":"ContainerStarted","Data":"58613b10c1f9355d876a9a5effe6d9dd3f410db92370fd6cb303f31a3e2d7f64"} Oct 31 23:21:30.935354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.935244 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:30.937340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.937317 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" event={"ID":"15bd6f6f-3f7e-46d2-87e1-2d4437233318","Type":"ContainerStarted","Data":"d7bb61de56151f81646839a04268d7dd53541c313e258365d103014c42ad5e72"} Oct 31 23:21:30.939742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.939719 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" event={"ID":"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa","Type":"ContainerStarted","Data":"33b03370dc51bb588203b26f74ca981d35efa15a889724c2f87a651d0a2a4cf9"} Oct 31 23:21:30.939885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.939871 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:30.941518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.941493 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" event={"ID":"7f3337f8-211c-4566-a0bd-521baacecd83","Type":"ContainerStarted","Data":"289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd"} Oct 31 23:21:30.941627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.941531 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" podUID="7f3337f8-211c-4566-a0bd-521baacecd83" containerName="revoke-keys" containerID="cri-o://289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd" gracePeriod=30 Oct 31 23:21:30.944271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.944256 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:21:30.944391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.944368 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2tbhj" event={"ID":"c8c42546-d756-4965-96c3-e3e05dd3d89b","Type":"ContainerStarted","Data":"5f4d2b109479b80baeef4f871189385aaefa735329b8616791d2a5e49a322336"} Oct 31 23:21:30.945884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.945865 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" event={"ID":"795bdc7a-da31-4a12-afad-cdc0ad0f7fed","Type":"ContainerStarted","Data":"8a1de1d51c2a93377f715c869cebbc87cc331d00a26a81d5495d7c5305d2f15b"} Oct 31 23:21:30.963698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.963642 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" podStartSLOduration=87.963624451 podStartE2EDuration="1m27.963624451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:30.955947896 +0000 UTC m=+1917.185502306" watchObservedRunningTime="2026-10-31 23:21:30.963624451 +0000 UTC m=+1917.193178853" Oct 31 23:21:30.990696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:30.990636 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" podStartSLOduration=8.99062242 podStartE2EDuration="8.99062242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:30.984873821 +0000 UTC m=+1917.214428230" watchObservedRunningTime="2026-10-31 23:21:30.99062242 +0000 UTC m=+1917.220177068" Oct 31 23:21:31.008264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.008209 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" podStartSLOduration=130.00819674 podStartE2EDuration="2m10.00819674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:31.004782544 +0000 UTC m=+1917.234336956" watchObservedRunningTime="2026-10-31 23:21:31.00819674 +0000 UTC m=+1917.237751150" Oct 31 23:21:31.037646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.037589 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" podStartSLOduration=11.037572976 podStartE2EDuration="11.037572976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:31.031533096 +0000 UTC m=+1917.261087508" watchObservedRunningTime="2026-10-31 23:21:31.037572976 +0000 UTC m=+1917.267127387" Oct 31 23:21:31.068192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.067834 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" podStartSLOduration=11.067818798 podStartE2EDuration="11.067818798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:31.060431111 +0000 UTC m=+1917.289985520" watchObservedRunningTime="2026-10-31 23:21:31.067818798 +0000 UTC m=+1917.297373216" Oct 31 23:21:31.095624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.095572 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" podStartSLOduration=101.095558912 podStartE2EDuration="1m41.095558912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:31.09235195 +0000 UTC m=+1917.321906361" watchObservedRunningTime="2026-10-31 23:21:31.095558912 +0000 UTC m=+1917.325113316" Oct 31 23:21:31.245333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.189512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:31.245333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.207936 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:31.376303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.376270 5043 patch_prober.go:29] interesting pod/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": read tcp 10.132.0.2:39626->10.132.0.179:15021: read: connection reset by peer" start-of-body= Oct 31 23:21:31.376448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.376335 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": read tcp 10.132.0.2:39626->10.132.0.179:15021: read: connection reset by peer" Oct 31 23:21:31.504527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.504450 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:31.606141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.606113 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:21:31.638000 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.637970 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca\") pod \"7f3337f8-211c-4566-a0bd-521baacecd83\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " Oct 31 23:21:31.638149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.638045 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca\") pod \"7f3337f8-211c-4566-a0bd-521baacecd83\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " Oct 31 23:21:31.638353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.638321 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg\" (UniqueName: \"kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg\") pod \"7f3337f8-211c-4566-a0bd-521baacecd83\" (UID: \"7f3337f8-211c-4566-a0bd-521baacecd83\") " Oct 31 23:21:31.638820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.638786 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca" pod "7f3337f8-211c-4566-a0bd-521baacecd83" (UID: "7f3337f8-211c-4566-a0bd-521baacecd83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:31.639079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.639060 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.641451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.641421 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca" pod "7f3337f8-211c-4566-a0bd-521baacecd83" (UID: "7f3337f8-211c-4566-a0bd-521baacecd83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:31.641777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.641752 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg" pod "7f3337f8-211c-4566-a0bd-521baacecd83" (UID: "7f3337f8-211c-4566-a0bd-521baacecd83"). InnerVolumeSpecName "kube-api-access-kr5kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:31.739684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.739655 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.739851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.739699 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.739851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.739721 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.739851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.739832 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.739960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.739926 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.740046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740032 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.740155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740142 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.740204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740191 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.740239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740224 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy\") pod \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\" (UID: \"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4\") " Oct 31 23:21:31.740708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740683 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:31.740900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740851 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:31.741016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740950 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3337f8-211c-4566-a0bd-521baacecd83-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.741142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.741023 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5kg\" (UniqueName: \"kubernetes.io/projected/7f3337f8-211c-4566-a0bd-521baacecd83-kube-api-access-kr5kg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.741142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740875 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:31.741142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.740962 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:31.741328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.741152 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:31.742709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.742684 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:31.742709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.742695 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:21:31.742840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.742724 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:31.742840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.742788 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c" pod "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" (UID: "95b4dacb-d2b9-4af2-8a5c-cd731ef023a4"). InnerVolumeSpecName "kube-api-access-kbh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:31.842594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842508 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842537 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842565 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842578 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842590 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842600 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbh8c\" (UniqueName: \"kubernetes.io/projected/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-kube-api-access-kbh8c\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842610 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842618 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.842893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.842627 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:31.938605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.938572 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06197912-3b3e-40aa-a19b-8025350bf804" path="/var/lib/kubelet/pods/06197912-3b3e-40aa-a19b-8025350bf804/volumes" Oct 31 23:21:31.939221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.939200 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" path="/var/lib/kubelet/pods/33eab6b9-e331-48da-bfbf-0d04e0baf215/volumes" Oct 31 23:21:31.939842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.939819 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" path="/var/lib/kubelet/pods/41cbfec0-981c-4be5-b984-0f1b8d71cc59/volumes" Oct 31 23:21:31.940650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.940630 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" path="/var/lib/kubelet/pods/47d31628-1d5e-4601-9b3f-3b514fed875e/volumes" Oct 31 23:21:31.941197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.941179 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" path="/var/lib/kubelet/pods/51479f0b-8c1b-472e-b4aa-80a088062549/volumes" Oct 31 23:21:31.941807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.941789 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" path="/var/lib/kubelet/pods/651744fd-77a3-431f-8f0e-a2f8509203f5/volumes" Oct 31 23:21:31.942601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.942583 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" path="/var/lib/kubelet/pods/842a6154-7c20-43e2-bc07-03a6cce2d71b/volumes" Oct 31 23:21:31.943097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.943078 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" path="/var/lib/kubelet/pods/b6a2e7de-778e-47c1-a35a-d4f066df6748/volumes" Oct 31 23:21:31.944154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.944130 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" path="/var/lib/kubelet/pods/f81afb95-ac07-4546-bea2-dfc33caf888d/volumes" Oct 31 23:21:31.958756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.958734 5043 generic.go:356] "Generic (PLEG): container finished" podID="28549d8a-85b5-4060-b623-be279fa8d14f" containerID="2758fbd6565169c3d9119226ad06d6bb67c0bf20bccb547ef5cc994e1cae9c60" exitCode=0 Oct 31 23:21:31.958852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.958815 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" event={"ID":"28549d8a-85b5-4060-b623-be279fa8d14f","Type":"ContainerDied","Data":"2758fbd6565169c3d9119226ad06d6bb67c0bf20bccb547ef5cc994e1cae9c60"} Oct 31 23:21:31.960518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.960497 5043 generic.go:356] "Generic (PLEG): container finished" podID="15bd6f6f-3f7e-46d2-87e1-2d4437233318" containerID="d7bb61de56151f81646839a04268d7dd53541c313e258365d103014c42ad5e72" exitCode=0 Oct 31 23:21:31.960616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.960572 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" event={"ID":"15bd6f6f-3f7e-46d2-87e1-2d4437233318","Type":"ContainerDied","Data":"d7bb61de56151f81646839a04268d7dd53541c313e258365d103014c42ad5e72"} Oct 31 23:21:31.962203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.962185 5043 generic.go:356] "Generic (PLEG): container finished" podID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerID="ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1" exitCode=0 Oct 31 23:21:31.962299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.962227 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" Oct 31 23:21:31.962299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.962262 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" event={"ID":"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4","Type":"ContainerDied","Data":"ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1"} Oct 31 23:21:31.962400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.962301 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc" event={"ID":"95b4dacb-d2b9-4af2-8a5c-cd731ef023a4","Type":"ContainerDied","Data":"fc44bf1bb657a7dd835299d22f9600066ee7502f96ef43f3619a64c59e8d3c53"} Oct 31 23:21:31.962400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.962329 5043 scope.go:122] "RemoveContainer" containerID="ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1" Oct 31 23:21:31.964076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.963888 5043 generic.go:356] "Generic (PLEG): container finished" podID="95df7923-e428-495b-a3a4-ea99cb1b013e" containerID="5c73ad771236e3c3375ef292a802d5c1364e3dd14277c0f5525afe2a22c6c991" exitCode=0 Oct 31 23:21:31.964076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.963962 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" event={"ID":"95df7923-e428-495b-a3a4-ea99cb1b013e","Type":"ContainerDied","Data":"5c73ad771236e3c3375ef292a802d5c1364e3dd14277c0f5525afe2a22c6c991"} Oct 31 23:21:31.966771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.966751 5043 generic.go:356] "Generic (PLEG): container finished" podID="7f3337f8-211c-4566-a0bd-521baacecd83" containerID="289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd" exitCode=0 Oct 31 23:21:31.966867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.966784 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" Oct 31 23:21:31.966867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.966828 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" event={"ID":"7f3337f8-211c-4566-a0bd-521baacecd83","Type":"ContainerDied","Data":"289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd"} Oct 31 23:21:31.966977 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.966865 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l" event={"ID":"7f3337f8-211c-4566-a0bd-521baacecd83","Type":"ContainerDied","Data":"9524bc4e8d318a5d1636a198a0e027d0980430f74baecf2ef55920b1ac1088b0"} Oct 31 23:21:31.968731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.968713 5043 generic.go:356] "Generic (PLEG): container finished" podID="74083906-8df2-4339-9148-e0d1f5c4de69" containerID="43d680ee9d1ab3f662447e423976d1a941a1f04d2a364349d62ca1799ea498a5" exitCode=0 Oct 31 23:21:31.968817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.968793 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" event={"ID":"74083906-8df2-4339-9148-e0d1f5c4de69","Type":"ContainerDied","Data":"43d680ee9d1ab3f662447e423976d1a941a1f04d2a364349d62ca1799ea498a5"} Oct 31 23:21:31.969236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.969216 5043 scope.go:122] "RemoveContainer" containerID="43d680ee9d1ab3f662447e423976d1a941a1f04d2a364349d62ca1799ea498a5" Oct 31 23:21:31.971113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.971089 5043 generic.go:356] "Generic (PLEG): container finished" podID="795bdc7a-da31-4a12-afad-cdc0ad0f7fed" containerID="8a1de1d51c2a93377f715c869cebbc87cc331d00a26a81d5495d7c5305d2f15b" exitCode=0 Oct 31 23:21:31.971294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.971270 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" event={"ID":"795bdc7a-da31-4a12-afad-cdc0ad0f7fed","Type":"ContainerDied","Data":"8a1de1d51c2a93377f715c869cebbc87cc331d00a26a81d5495d7c5305d2f15b"} Oct 31 23:21:31.972845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.972826 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:31.973679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.973660 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:21:31.977645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.977627 5043 scope.go:122] "RemoveContainer" containerID="8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12" Oct 31 23:21:31.977760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.977731 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:31.977829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.977813 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:31.977878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.977859 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:31.993389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.993369 5043 scope.go:122] "RemoveContainer" containerID="ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1" Oct 31 23:21:31.993793 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:31.993669 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1\": container with ID starting with ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1 not found: ID does not exist" containerID="ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1" Oct 31 23:21:31.993793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.993704 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1"} err="failed to get container status \"ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1\": rpc error: code = NotFound desc = could not find container \"ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1\": container with ID starting with ffb2ac1f6c9f8882e5a17d367305b8e4c53541c259a93fd3e5c162e2697964a1 not found: ID does not exist" Oct 31 23:21:31.993793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.993725 5043 scope.go:122] "RemoveContainer" containerID="8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12" Oct 31 23:21:31.994136 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:31.994111 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12\": container with ID starting with 8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12 not found: ID does not exist" containerID="8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12" Oct 31 23:21:31.994136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.994143 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12"} err="failed to get container status \"8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12\": rpc error: code = NotFound desc = could not find container \"8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12\": container with ID starting with 8f2f35ef4052f406d93574f8867e24ec540ed65cab2379f6dc2494a2c2f23d12 not found: ID does not exist" Oct 31 23:21:31.994136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:31.994164 5043 scope.go:122] "RemoveContainer" containerID="289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd" Oct 31 23:21:32.005177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.005152 5043 scope.go:122] "RemoveContainer" containerID="289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd" Oct 31 23:21:32.005461 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:32.005442 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd\": container with ID starting with 289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd not found: ID does not exist" containerID="289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd" Oct 31 23:21:32.005540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.005463 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd"} err="failed to get container status \"289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd\": rpc error: code = NotFound desc = could not find container \"289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd\": container with ID starting with 289a37186498a9e82e10a59ec5cff9e50ac81c718305ae740690b6d4138700dd not found: ID does not exist" Oct 31 23:21:32.029522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.024895 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l"] Oct 31 23:21:32.032087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.032066 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-8th9l"] Oct 31 23:21:32.207924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.207836 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc"] Oct 31 23:21:32.213380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.213356 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2294a7-openshift-default-84889df665-58qrc"] Oct 31 23:21:32.364854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.364809 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" containerID="cri-o://76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963" gracePeriod=43 Oct 31 23:21:32.365011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.364811 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" containerID="cri-o://d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795" gracePeriod=45 Oct 31 23:21:32.365011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.364921 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" containerID="cri-o://16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc" gracePeriod=45 Oct 31 23:21:32.533031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.532960 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:32.606303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.606256 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": dial tcp 10.132.0.110:8000: connect: connection refused" Oct 31 23:21:32.709433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.709356 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:21:32.733316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.733255 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:21:32.749443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.749405 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:21:32.862481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862449 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache\") pod \"013ef212-2c66-476e-91fb-38ac1d259d38\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " Oct 31 23:21:32.862690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862511 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir\") pod \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " Oct 31 23:21:32.862690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862540 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs\") pod \"3142d597-b170-4c68-8b9d-869fb8bda3b8\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " Oct 31 23:21:32.862690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862616 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2\" (UniqueName: \"kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2\") pod \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " Oct 31 23:21:32.862848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862691 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897\" (UniqueName: \"kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897\") pod \"3142d597-b170-4c68-8b9d-869fb8bda3b8\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " Oct 31 23:21:32.862848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862802 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs\") pod \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " Oct 31 23:21:32.862848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862817 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache" pod "013ef212-2c66-476e-91fb-38ac1d259d38" (UID: "013ef212-2c66-476e-91fb-38ac1d259d38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.862986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862852 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache\") pod \"3142d597-b170-4c68-8b9d-869fb8bda3b8\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " Oct 31 23:21:32.862986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862865 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir" pod "6407e3cb-e7ef-48fa-b7fb-320aaff5408e" (UID: "6407e3cb-e7ef-48fa-b7fb-320aaff5408e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.862986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862885 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home\") pod \"013ef212-2c66-476e-91fb-38ac1d259d38\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " Oct 31 23:21:32.862986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862908 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l\" (UniqueName: \"kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l\") pod \"013ef212-2c66-476e-91fb-38ac1d259d38\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " Oct 31 23:21:32.862986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862929 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache\") pod \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " Oct 31 23:21:32.862986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.862975 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home\") pod \"3142d597-b170-4c68-8b9d-869fb8bda3b8\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " Oct 31 23:21:32.863280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863045 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs\" (UniqueName: \"kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs\") pod \"013ef212-2c66-476e-91fb-38ac1d259d38\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " Oct 31 23:21:32.863280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863096 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home\") pod \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " Oct 31 23:21:32.863280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863129 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir\") pod \"3142d597-b170-4c68-8b9d-869fb8bda3b8\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " Oct 31 23:21:32.863280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863183 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm\") pod \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\" (UID: \"6407e3cb-e7ef-48fa-b7fb-320aaff5408e\") " Oct 31 23:21:32.863280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863214 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm\") pod \"013ef212-2c66-476e-91fb-38ac1d259d38\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " Oct 31 23:21:32.863280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863256 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home" pod "013ef212-2c66-476e-91fb-38ac1d259d38" (UID: "013ef212-2c66-476e-91fb-38ac1d259d38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.863609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863281 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm\") pod \"3142d597-b170-4c68-8b9d-869fb8bda3b8\" (UID: \"3142d597-b170-4c68-8b9d-869fb8bda3b8\") " Oct 31 23:21:32.863609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863287 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache" pod "6407e3cb-e7ef-48fa-b7fb-320aaff5408e" (UID: "6407e3cb-e7ef-48fa-b7fb-320aaff5408e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.863609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863325 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir\") pod \"013ef212-2c66-476e-91fb-38ac1d259d38\" (UID: \"013ef212-2c66-476e-91fb-38ac1d259d38\") " Oct 31 23:21:32.863770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863751 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home" pod "3142d597-b170-4c68-8b9d-869fb8bda3b8" (UID: "3142d597-b170-4c68-8b9d-869fb8bda3b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.863962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.863939 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir" pod "013ef212-2c66-476e-91fb-38ac1d259d38" (UID: "013ef212-2c66-476e-91fb-38ac1d259d38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.864106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864082 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir" pod "3142d597-b170-4c68-8b9d-869fb8bda3b8" (UID: "3142d597-b170-4c68-8b9d-869fb8bda3b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.864274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864255 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864281 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864294 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864307 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864350 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864363 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864376 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.864582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864402 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache" pod "3142d597-b170-4c68-8b9d-869fb8bda3b8" (UID: "3142d597-b170-4c68-8b9d-869fb8bda3b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.864754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.864601 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home" pod "6407e3cb-e7ef-48fa-b7fb-320aaff5408e" (UID: "6407e3cb-e7ef-48fa-b7fb-320aaff5408e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.866363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.866334 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs" pod "6407e3cb-e7ef-48fa-b7fb-320aaff5408e" (UID: "6407e3cb-e7ef-48fa-b7fb-320aaff5408e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:32.866840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.866803 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2" pod "6407e3cb-e7ef-48fa-b7fb-320aaff5408e" (UID: "6407e3cb-e7ef-48fa-b7fb-320aaff5408e"). InnerVolumeSpecName "kube-api-access-45lq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:32.868337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.868301 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897" pod "3142d597-b170-4c68-8b9d-869fb8bda3b8" (UID: "3142d597-b170-4c68-8b9d-869fb8bda3b8"). InnerVolumeSpecName "kube-api-access-5p897". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:32.869833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.869808 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm" pod "3142d597-b170-4c68-8b9d-869fb8bda3b8" (UID: "3142d597-b170-4c68-8b9d-869fb8bda3b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.870904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.870882 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm" pod "6407e3cb-e7ef-48fa-b7fb-320aaff5408e" (UID: "6407e3cb-e7ef-48fa-b7fb-320aaff5408e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.870984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.870881 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs" pod "013ef212-2c66-476e-91fb-38ac1d259d38" (UID: "013ef212-2c66-476e-91fb-38ac1d259d38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:32.871142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.871116 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs" pod "3142d597-b170-4c68-8b9d-869fb8bda3b8" (UID: "3142d597-b170-4c68-8b9d-869fb8bda3b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:32.871200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.871146 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l" pod "013ef212-2c66-476e-91fb-38ac1d259d38" (UID: "013ef212-2c66-476e-91fb-38ac1d259d38"). InnerVolumeSpecName "kube-api-access-jdg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:32.872392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.872366 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm" pod "013ef212-2c66-476e-91fb-38ac1d259d38" (UID: "013ef212-2c66-476e-91fb-38ac1d259d38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:32.966301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966259 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966305 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966321 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdg6l\" (UniqueName: \"kubernetes.io/projected/013ef212-2c66-476e-91fb-38ac1d259d38-kube-api-access-jdg6l\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966335 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013ef212-2c66-476e-91fb-38ac1d259d38-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966373 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966390 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966404 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013ef212-2c66-476e-91fb-38ac1d259d38-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966415 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3142d597-b170-4c68-8b9d-869fb8bda3b8-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966425 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3142d597-b170-4c68-8b9d-869fb8bda3b8-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966438 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lq2\" (UniqueName: \"kubernetes.io/projected/6407e3cb-e7ef-48fa-b7fb-320aaff5408e-kube-api-access-45lq2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.966692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.966453 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p897\" (UniqueName: \"kubernetes.io/projected/3142d597-b170-4c68-8b9d-869fb8bda3b8-kube-api-access-5p897\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:32.984271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.984200 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" event={"ID":"74083906-8df2-4339-9148-e0d1f5c4de69","Type":"ContainerStarted","Data":"4211256ebe40147676ce3dc6bd532c51a736f550e2e6f98e987f48f8c8b361e7"} Oct 31 23:21:32.985501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.985478 5043 generic.go:356] "Generic (PLEG): container finished" podID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerID="16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc" exitCode=0 Oct 31 23:21:32.985612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.985513 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" Oct 31 23:21:32.985612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.985568 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" event={"ID":"6407e3cb-e7ef-48fa-b7fb-320aaff5408e","Type":"ContainerDied","Data":"16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc"} Oct 31 23:21:32.985612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.985600 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" event={"ID":"6407e3cb-e7ef-48fa-b7fb-320aaff5408e","Type":"ContainerDied","Data":"30a6d0f7ea7404f3e34a98ebd9e0a5f2c5f9b9da769fa2c9d41ba967a5075856"} Oct 31 23:21:32.985749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.985620 5043 scope.go:122] "RemoveContainer" containerID="16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc" Oct 31 23:21:32.987029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.987007 5043 generic.go:356] "Generic (PLEG): container finished" podID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerID="d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795" exitCode=0 Oct 31 23:21:32.987109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.987053 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" Oct 31 23:21:32.987109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.987098 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" event={"ID":"3142d597-b170-4c68-8b9d-869fb8bda3b8","Type":"ContainerDied","Data":"d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795"} Oct 31 23:21:32.987220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.987131 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m" event={"ID":"3142d597-b170-4c68-8b9d-869fb8bda3b8","Type":"ContainerDied","Data":"7f986c5490b9fdba00312659f292ed6187d497cccb46099e4347bb78e45f83de"} Oct 31 23:21:32.997765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.997731 5043 generic.go:356] "Generic (PLEG): container finished" podID="013ef212-2c66-476e-91fb-38ac1d259d38" containerID="76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963" exitCode=0 Oct 31 23:21:32.998271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.998210 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" Oct 31 23:21:32.998969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.998895 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" event={"ID":"013ef212-2c66-476e-91fb-38ac1d259d38","Type":"ContainerDied","Data":"76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963"} Oct 31 23:21:32.998969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:32.998935 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6" event={"ID":"013ef212-2c66-476e-91fb-38ac1d259d38","Type":"ContainerDied","Data":"5dbb668beac83d30ec01ad2ec22f97cd6db65e2c39ea09b3794d58ac73982254"} Oct 31 23:21:33.014088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.014069 5043 scope.go:122] "RemoveContainer" containerID="16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc" Oct 31 23:21:33.014419 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:33.014388 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc\": container with ID starting with 16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc not found: ID does not exist" containerID="16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc" Oct 31 23:21:33.014509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.014429 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc"} err="failed to get container status \"16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc\": rpc error: code = NotFound desc = could not find container \"16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc\": container with ID starting with 16d3b06a079e3fb6705f4397988651d171c056f9bab40f8a4c8b0cc1f7fbc8cc not found: ID does not exist" Oct 31 23:21:33.014509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.014455 5043 scope.go:122] "RemoveContainer" containerID="d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795" Oct 31 23:21:33.055324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.055288 5043 scope.go:122] "RemoveContainer" containerID="d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795" Oct 31 23:21:33.055941 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:33.055912 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795\": container with ID starting with d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795 not found: ID does not exist" containerID="d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795" Oct 31 23:21:33.055996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.055964 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795"} err="failed to get container status \"d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795\": rpc error: code = NotFound desc = could not find container \"d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795\": container with ID starting with d428c72cb3a34988cc9fe07f53e2afdae44ebad9f6ed68dc68484e4a9c7b2795 not found: ID does not exist" Oct 31 23:21:33.055996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.055993 5043 scope.go:122] "RemoveContainer" containerID="76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963" Oct 31 23:21:33.081034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.080103 5043 scope.go:122] "RemoveContainer" containerID="76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963" Oct 31 23:21:33.084774 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:33.083091 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963\": container with ID starting with 76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963 not found: ID does not exist" containerID="76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963" Oct 31 23:21:33.084774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.083139 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963"} err="failed to get container status \"76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963\": rpc error: code = NotFound desc = could not find container \"76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963\": container with ID starting with 76a723dc02f17ac70a84070db5e703ca12e554fc8302c21d059044c469ae9963 not found: ID does not exist" Oct 31 23:21:33.145525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.145501 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6"] Oct 31 23:21:33.159024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.158998 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-w0-a16dfb-kserve-79b9db64b9-pmdk6"] Oct 31 23:21:33.177145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.177103 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m"] Oct 31 23:21:33.183336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.183309 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/distinct-2-w0-a16dfb-kserve-668cb947f4-vfz5m"] Oct 31 23:21:33.210611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.210589 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4"] Oct 31 23:21:33.232064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.232035 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4"] Oct 31 23:21:33.240405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.240342 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:33.307858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.307111 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:33.314765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.314738 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:33.320922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.320897 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:33.375086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.375051 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca\") pod \"95df7923-e428-495b-a3a4-ea99cb1b013e\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " Oct 31 23:21:33.375375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.375354 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj\" (UniqueName: \"kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj\") pod \"95df7923-e428-495b-a3a4-ea99cb1b013e\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " Oct 31 23:21:33.375441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.375412 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca\") pod \"95df7923-e428-495b-a3a4-ea99cb1b013e\" (UID: \"95df7923-e428-495b-a3a4-ea99cb1b013e\") " Oct 31 23:21:33.375488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.375467 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca" pod "95df7923-e428-495b-a3a4-ea99cb1b013e" (UID: "95df7923-e428-495b-a3a4-ea99cb1b013e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:33.376314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.376286 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.377598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.377524 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca" pod "95df7923-e428-495b-a3a4-ea99cb1b013e" (UID: "95df7923-e428-495b-a3a4-ea99cb1b013e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:33.378055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.378029 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj" pod "95df7923-e428-495b-a3a4-ea99cb1b013e" (UID: "95df7923-e428-495b-a3a4-ea99cb1b013e"). InnerVolumeSpecName "kube-api-access-dcktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:33.434902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.434867 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:21:33.440888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.440858 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:21:33.477454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477422 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca\") pod \"28549d8a-85b5-4060-b623-be279fa8d14f\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " Oct 31 23:21:33.477639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477512 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca\") pod \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " Oct 31 23:21:33.477639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477606 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w\" (UniqueName: \"kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w\") pod \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " Oct 31 23:21:33.477770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477649 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca\") pod \"28549d8a-85b5-4060-b623-be279fa8d14f\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " Oct 31 23:21:33.477826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477805 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca" pod "28549d8a-85b5-4060-b623-be279fa8d14f" (UID: "28549d8a-85b5-4060-b623-be279fa8d14f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:33.477876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477861 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca\") pod \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\" (UID: \"795bdc7a-da31-4a12-afad-cdc0ad0f7fed\") " Oct 31 23:21:33.477923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.477880 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca" pod "795bdc7a-da31-4a12-afad-cdc0ad0f7fed" (UID: "795bdc7a-da31-4a12-afad-cdc0ad0f7fed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:33.478031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478013 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca\") pod \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " Oct 31 23:21:33.478093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478044 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca\") pod \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " Oct 31 23:21:33.478150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478100 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g\" (UniqueName: \"kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g\") pod \"28549d8a-85b5-4060-b623-be279fa8d14f\" (UID: \"28549d8a-85b5-4060-b623-be279fa8d14f\") " Oct 31 23:21:33.478206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478168 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl\" (UniqueName: \"kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl\") pod \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\" (UID: \"15bd6f6f-3f7e-46d2-87e1-2d4437233318\") " Oct 31 23:21:33.478535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478508 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca" pod "15bd6f6f-3f7e-46d2-87e1-2d4437233318" (UID: "15bd6f6f-3f7e-46d2-87e1-2d4437233318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:33.478964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478936 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.479075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478968 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.479075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478983 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcktj\" (UniqueName: \"kubernetes.io/projected/95df7923-e428-495b-a3a4-ea99cb1b013e-kube-api-access-dcktj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.479075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.478997 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95df7923-e428-495b-a3a4-ea99cb1b013e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.479075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.479010 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.480695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.480602 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca" pod "795bdc7a-da31-4a12-afad-cdc0ad0f7fed" (UID: "795bdc7a-da31-4a12-afad-cdc0ad0f7fed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:33.480695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.480645 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w" pod "795bdc7a-da31-4a12-afad-cdc0ad0f7fed" (UID: "795bdc7a-da31-4a12-afad-cdc0ad0f7fed"). InnerVolumeSpecName "kube-api-access-rk45w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:33.480695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.480671 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca" pod "28549d8a-85b5-4060-b623-be279fa8d14f" (UID: "28549d8a-85b5-4060-b623-be279fa8d14f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:33.480899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.480691 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca" pod "15bd6f6f-3f7e-46d2-87e1-2d4437233318" (UID: "15bd6f6f-3f7e-46d2-87e1-2d4437233318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:33.481941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.481917 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl" pod "15bd6f6f-3f7e-46d2-87e1-2d4437233318" (UID: "15bd6f6f-3f7e-46d2-87e1-2d4437233318"). InnerVolumeSpecName "kube-api-access-d72xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:33.482143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.482126 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g" pod "28549d8a-85b5-4060-b623-be279fa8d14f" (UID: "28549d8a-85b5-4060-b623-be279fa8d14f"). InnerVolumeSpecName "kube-api-access-klr5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:33.520534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.520504 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:33.580539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.580515 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15bd6f6f-3f7e-46d2-87e1-2d4437233318-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.580846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.580539 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klr5g\" (UniqueName: \"kubernetes.io/projected/28549d8a-85b5-4060-b623-be279fa8d14f-kube-api-access-klr5g\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.580846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.580584 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d72xl\" (UniqueName: \"kubernetes.io/projected/15bd6f6f-3f7e-46d2-87e1-2d4437233318-kube-api-access-d72xl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.580846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.580593 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk45w\" (UniqueName: \"kubernetes.io/projected/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-kube-api-access-rk45w\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.580846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.580602 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28549d8a-85b5-4060-b623-be279fa8d14f-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.580846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.580611 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/795bdc7a-da31-4a12-afad-cdc0ad0f7fed-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:33.633470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.633436 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a16dfb/simulator-w0-a16dfb-kserve-748f84879-rmvq4" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 23:21:33.721344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.721301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:33.734152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.734091 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" containerID="cri-o://a56c04ce6a42b4c3ab48ad4ecb4243c63177ccdfb0d809e8f43ae2151b95cbd1" gracePeriod=45 Oct 31 23:21:33.736986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.736955 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" containerID="cri-o://2c9638959b2e675c1e2d903c50f23d0278240b35e1e1e31c7331c16adcef7f9e" gracePeriod=45 Oct 31 23:21:33.754204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.742619 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" containerID="cri-o://4cdb91259c2612b02b50e600a37977a9094e8d130a9fae523b0b2a441a94883b" gracePeriod=45 Oct 31 23:21:33.754204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.744024 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" containerID="cri-o://897efee68277d18102448bf405190583a7482170251dcd805e74f056d02d63a0" gracePeriod=45 Oct 31 23:21:33.954004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.947756 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" path="/var/lib/kubelet/pods/013ef212-2c66-476e-91fb-38ac1d259d38/volumes" Oct 31 23:21:33.954004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.948887 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" path="/var/lib/kubelet/pods/3142d597-b170-4c68-8b9d-869fb8bda3b8/volumes" Oct 31 23:21:33.954004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.949789 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" path="/var/lib/kubelet/pods/6407e3cb-e7ef-48fa-b7fb-320aaff5408e/volumes" Oct 31 23:21:33.954004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.951792 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3337f8-211c-4566-a0bd-521baacecd83" path="/var/lib/kubelet/pods/7f3337f8-211c-4566-a0bd-521baacecd83/volumes" Oct 31 23:21:33.954004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:33.952918 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" path="/var/lib/kubelet/pods/95b4dacb-d2b9-4af2-8a5c-cd731ef023a4/volumes" Oct 31 23:21:34.017218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.017184 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" event={"ID":"795bdc7a-da31-4a12-afad-cdc0ad0f7fed","Type":"ContainerDied","Data":"8e199f978f4ac490a496a8ea8ab0e36b6b1b4b9bc6532a3fbd08312960dcf842"} Oct 31 23:21:34.017452 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.017434 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e199f978f4ac490a496a8ea8ab0e36b6b1b4b9bc6532a3fbd08312960dcf842" Oct 31 23:21:34.017676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.017660 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g" Oct 31 23:21:34.079609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.075676 5043 generic.go:356] "Generic (PLEG): container finished" podID="f23336de-2435-41dd-a86e-629d91536e27" containerID="a56c04ce6a42b4c3ab48ad4ecb4243c63177ccdfb0d809e8f43ae2151b95cbd1" exitCode=0 Oct 31 23:21:34.084178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.079702 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" event={"ID":"f23336de-2435-41dd-a86e-629d91536e27","Type":"ContainerDied","Data":"a56c04ce6a42b4c3ab48ad4ecb4243c63177ccdfb0d809e8f43ae2151b95cbd1"} Oct 31 23:21:34.096573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.095915 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:34.099882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.099856 5043 generic.go:356] "Generic (PLEG): container finished" podID="d9bbad83-e55f-4905-941c-cc28506303ed" containerID="897efee68277d18102448bf405190583a7482170251dcd805e74f056d02d63a0" exitCode=0 Oct 31 23:21:34.100154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.100130 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" event={"ID":"d9bbad83-e55f-4905-941c-cc28506303ed","Type":"ContainerDied","Data":"897efee68277d18102448bf405190583a7482170251dcd805e74f056d02d63a0"} Oct 31 23:21:34.164796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.164761 5043 generic.go:356] "Generic (PLEG): container finished" podID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerID="2c9638959b2e675c1e2d903c50f23d0278240b35e1e1e31c7331c16adcef7f9e" exitCode=0 Oct 31 23:21:34.165089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.165059 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" event={"ID":"5a0ded48-b028-4b3b-99ac-665ba58e14f6","Type":"ContainerDied","Data":"2c9638959b2e675c1e2d903c50f23d0278240b35e1e1e31c7331c16adcef7f9e"} Oct 31 23:21:34.167689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.167666 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" event={"ID":"28549d8a-85b5-4060-b623-be279fa8d14f","Type":"ContainerDied","Data":"524b01ff1d807fdc918747afe744812d05a5c26f2dcbe294165bed49d2fb89df"} Oct 31 23:21:34.167839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.167825 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524b01ff1d807fdc918747afe744812d05a5c26f2dcbe294165bed49d2fb89df" Oct 31 23:21:34.168064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.168049 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct" Oct 31 23:21:34.171355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.171320 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" event={"ID":"15bd6f6f-3f7e-46d2-87e1-2d4437233318","Type":"ContainerDied","Data":"b721301d75fe919f6f08fe8cc88f51f3a9e7e429ee0d73f98c954f61c08a5d94"} Oct 31 23:21:34.171463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.171377 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b721301d75fe919f6f08fe8cc88f51f3a9e7e429ee0d73f98c954f61c08a5d94" Oct 31 23:21:34.171521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.171468 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79" Oct 31 23:21:34.175538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.175514 5043 generic.go:356] "Generic (PLEG): container finished" podID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerID="4cdb91259c2612b02b50e600a37977a9094e8d130a9fae523b0b2a441a94883b" exitCode=0 Oct 31 23:21:34.175929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.175906 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" event={"ID":"3ac6cb61-cfe6-4efc-82db-95b60b80b33a","Type":"ContainerDied","Data":"4cdb91259c2612b02b50e600a37977a9094e8d130a9fae523b0b2a441a94883b"} Oct 31 23:21:34.181848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.181827 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" event={"ID":"95df7923-e428-495b-a3a4-ea99cb1b013e","Type":"ContainerDied","Data":"517d2682702a21e4caf1cff07ec2f18830a4204c0ba15568137143b543e6f5cd"} Oct 31 23:21:34.181981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.181966 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517d2682702a21e4caf1cff07ec2f18830a4204c0ba15568137143b543e6f5cd" Oct 31 23:21:34.182182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.182169 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46" Oct 31 23:21:34.183152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.183137 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:21:34.185783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.185760 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:21:34.200744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.200725 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:21:34.272563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.271051 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:21:34.282721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.282694 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:21:34.287189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.287168 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399167 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home\") pod \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399241 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs\") pod \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399282 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856\" (UniqueName: \"kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856\") pod \"d9bbad83-e55f-4905-941c-cc28506303ed\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399325 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache\") pod \"f23336de-2435-41dd-a86e-629d91536e27\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399384 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir\") pod \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399412 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5\" (UniqueName: \"kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5\") pod \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399498 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm\") pod \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399532 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm\") pod \"f23336de-2435-41dd-a86e-629d91536e27\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " Oct 31 23:21:34.399692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399622 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs\") pod \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399677 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir\") pod \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399810 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs\") pod \"d9bbad83-e55f-4905-941c-cc28506303ed\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399903 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs\" (UniqueName: \"kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs\") pod \"f23336de-2435-41dd-a86e-629d91536e27\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399948 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir\") pod \"f23336de-2435-41dd-a86e-629d91536e27\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.399997 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache\") pod \"d9bbad83-e55f-4905-941c-cc28506303ed\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400032 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache\") pod \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400104 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm\") pod \"d9bbad83-e55f-4905-941c-cc28506303ed\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400139 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir\") pod \"d9bbad83-e55f-4905-941c-cc28506303ed\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400174 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home\") pod \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400235 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm\") pod \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\" (UID: \"5a0ded48-b028-4b3b-99ac-665ba58e14f6\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400280 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache\") pod \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400325 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6\" (UniqueName: \"kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6\") pod \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\" (UID: \"3ac6cb61-cfe6-4efc-82db-95b60b80b33a\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400379 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home\") pod \"d9bbad83-e55f-4905-941c-cc28506303ed\" (UID: \"d9bbad83-e55f-4905-941c-cc28506303ed\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400410 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home\") pod \"f23336de-2435-41dd-a86e-629d91536e27\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " Oct 31 23:21:34.401706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.400462 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778\" (UniqueName: \"kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778\") pod \"f23336de-2435-41dd-a86e-629d91536e27\" (UID: \"f23336de-2435-41dd-a86e-629d91536e27\") " Oct 31 23:21:34.405764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.402012 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home" pod "5a0ded48-b028-4b3b-99ac-665ba58e14f6" (UID: "5a0ded48-b028-4b3b-99ac-665ba58e14f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.405764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.403232 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir" pod "3ac6cb61-cfe6-4efc-82db-95b60b80b33a" (UID: "3ac6cb61-cfe6-4efc-82db-95b60b80b33a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.405764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.403410 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir" pod "f23336de-2435-41dd-a86e-629d91536e27" (UID: "f23336de-2435-41dd-a86e-629d91536e27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.405764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.404701 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache" pod "f23336de-2435-41dd-a86e-629d91536e27" (UID: "f23336de-2435-41dd-a86e-629d91536e27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.407376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.407123 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir" pod "5a0ded48-b028-4b3b-99ac-665ba58e14f6" (UID: "5a0ded48-b028-4b3b-99ac-665ba58e14f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.408755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.408477 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856" pod "d9bbad83-e55f-4905-941c-cc28506303ed" (UID: "d9bbad83-e55f-4905-941c-cc28506303ed"). InnerVolumeSpecName "kube-api-access-qc856". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:34.409671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.409361 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5" pod "5a0ded48-b028-4b3b-99ac-665ba58e14f6" (UID: "5a0ded48-b028-4b3b-99ac-665ba58e14f6"). InnerVolumeSpecName "kube-api-access-mhzf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.411442 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm" pod "d9bbad83-e55f-4905-941c-cc28506303ed" (UID: "d9bbad83-e55f-4905-941c-cc28506303ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.411617 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs" pod "f23336de-2435-41dd-a86e-629d91536e27" (UID: "f23336de-2435-41dd-a86e-629d91536e27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.411715 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache" pod "d9bbad83-e55f-4905-941c-cc28506303ed" (UID: "d9bbad83-e55f-4905-941c-cc28506303ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.411924 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache" pod "5a0ded48-b028-4b3b-99ac-665ba58e14f6" (UID: "5a0ded48-b028-4b3b-99ac-665ba58e14f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.411935 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir" pod "d9bbad83-e55f-4905-941c-cc28506303ed" (UID: "d9bbad83-e55f-4905-941c-cc28506303ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.412110 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home" pod "d9bbad83-e55f-4905-941c-cc28506303ed" (UID: "d9bbad83-e55f-4905-941c-cc28506303ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.412143 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home" pod "f23336de-2435-41dd-a86e-629d91536e27" (UID: "f23336de-2435-41dd-a86e-629d91536e27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.412410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.412292 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache" pod "3ac6cb61-cfe6-4efc-82db-95b60b80b33a" (UID: "3ac6cb61-cfe6-4efc-82db-95b60b80b33a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.413267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.413051 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs" pod "3ac6cb61-cfe6-4efc-82db-95b60b80b33a" (UID: "3ac6cb61-cfe6-4efc-82db-95b60b80b33a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:34.413267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.413231 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home" pod "3ac6cb61-cfe6-4efc-82db-95b60b80b33a" (UID: "3ac6cb61-cfe6-4efc-82db-95b60b80b33a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.414386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.414354 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm" pod "3ac6cb61-cfe6-4efc-82db-95b60b80b33a" (UID: "3ac6cb61-cfe6-4efc-82db-95b60b80b33a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.415234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.415187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778" pod "f23336de-2435-41dd-a86e-629d91536e27" (UID: "f23336de-2435-41dd-a86e-629d91536e27"). InnerVolumeSpecName "kube-api-access-pq778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:34.415918 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.415893 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6" pod "3ac6cb61-cfe6-4efc-82db-95b60b80b33a" (UID: "3ac6cb61-cfe6-4efc-82db-95b60b80b33a"). InnerVolumeSpecName "kube-api-access-lsqc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:34.416061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.416035 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs" pod "5a0ded48-b028-4b3b-99ac-665ba58e14f6" (UID: "5a0ded48-b028-4b3b-99ac-665ba58e14f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:34.416467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.416450 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm" pod "5a0ded48-b028-4b3b-99ac-665ba58e14f6" (UID: "5a0ded48-b028-4b3b-99ac-665ba58e14f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.417814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.417789 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm" pod "f23336de-2435-41dd-a86e-629d91536e27" (UID: "f23336de-2435-41dd-a86e-629d91536e27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:21:34.418481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.418455 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs" pod "d9bbad83-e55f-4905-941c-cc28506303ed" (UID: "d9bbad83-e55f-4905-941c-cc28506303ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:21:34.503074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503042 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bbad83-e55f-4905-941c-cc28506303ed-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503068 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23336de-2435-41dd-a86e-629d91536e27-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503080 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503090 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503102 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503111 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503119 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503127 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503134 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503142 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503152 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqc6\" (UniqueName: \"kubernetes.io/projected/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-kube-api-access-lsqc6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503162 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9bbad83-e55f-4905-941c-cc28506303ed-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503171 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503179 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq778\" (UniqueName: \"kubernetes.io/projected/f23336de-2435-41dd-a86e-629d91536e27-kube-api-access-pq778\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503187 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503196 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503204 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc856\" (UniqueName: \"kubernetes.io/projected/d9bbad83-e55f-4905-941c-cc28506303ed-kube-api-access-qc856\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503213 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503222 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0ded48-b028-4b3b-99ac-665ba58e14f6-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503231 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzf5\" (UniqueName: \"kubernetes.io/projected/5a0ded48-b028-4b3b-99ac-665ba58e14f6-kube-api-access-mhzf5\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503239 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503246 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23336de-2435-41dd-a86e-629d91536e27-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503254 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.503365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.503273 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac6cb61-cfe6-4efc-82db-95b60b80b33a-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:34.915482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.915446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:34.982251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.982219 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl"] Oct 31 23:21:34.983077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983061 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerName="istio-proxy" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983079 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerName="istio-proxy" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983096 5043 cpu_manager.go:414] "containerMap: removing container" podUID="795bdc7a-da31-4a12-afad-cdc0ad0f7fed" containerName="revoke-keys" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983101 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795bdc7a-da31-4a12-afad-cdc0ad0f7fed" containerName="revoke-keys" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983113 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983119 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983127 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983132 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" Oct 31 23:21:34.983142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983140 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983145 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983152 5043 cpu_manager.go:414] "containerMap: removing container" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983157 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983166 5043 cpu_manager.go:414] "containerMap: removing container" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983170 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983186 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983190 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983206 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983213 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983225 5043 cpu_manager.go:414] "containerMap: removing container" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983229 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983240 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983245 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983251 5043 cpu_manager.go:414] "containerMap: removing container" podUID="95df7923-e428-495b-a3a4-ea99cb1b013e" containerName="revoke-keys" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983255 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95df7923-e428-495b-a3a4-ea99cb1b013e" containerName="revoke-keys" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983263 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983267 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983281 5043 cpu_manager.go:414] "containerMap: removing container" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983285 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983296 5043 cpu_manager.go:414] "containerMap: removing container" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983301 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983309 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983314 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983325 5043 cpu_manager.go:414] "containerMap: removing container" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983330 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983335 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7f3337f8-211c-4566-a0bd-521baacecd83" containerName="revoke-keys" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983339 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3337f8-211c-4566-a0bd-521baacecd83" containerName="revoke-keys" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983348 5043 cpu_manager.go:414] "containerMap: removing container" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983353 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983371 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983376 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983383 5043 cpu_manager.go:414] "containerMap: removing container" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" Oct 31 23:21:34.983376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983388 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983396 5043 cpu_manager.go:414] "containerMap: removing container" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983400 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983411 5043 cpu_manager.go:414] "containerMap: removing container" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983416 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983423 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983427 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983440 5043 cpu_manager.go:414] "containerMap: removing container" podUID="28549d8a-85b5-4060-b623-be279fa8d14f" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983444 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28549d8a-85b5-4060-b623-be279fa8d14f" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983450 5043 cpu_manager.go:414] "containerMap: removing container" podUID="15bd6f6f-3f7e-46d2-87e1-2d4437233318" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983455 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bd6f6f-3f7e-46d2-87e1-2d4437233318" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983470 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983475 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983487 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983491 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983613 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983623 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81afb95-ac07-4546-bea2-dfc33caf888d" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983630 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="33eab6b9-e331-48da-bfbf-0d04e0baf215" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983636 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cbfec0-981c-4be5-b984-0f1b8d71cc59" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983644 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983649 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6407e3cb-e7ef-48fa-b7fb-320aaff5408e" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983654 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="28549d8a-85b5-4060-b623-be279fa8d14f" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983659 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bd6f6f-3f7e-46d2-87e1-2d4437233318" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983666 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983672 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983677 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3142d597-b170-4c68-8b9d-869fb8bda3b8" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983682 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a2e7de-778e-47c1-a35a-d4f066df6748" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983688 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="795bdc7a-da31-4a12-afad-cdc0ad0f7fed" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983695 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983701 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983708 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983715 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="013ef212-2c66-476e-91fb-38ac1d259d38" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983721 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="51479f0b-8c1b-472e-b4aa-80a088062549" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983728 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" containerName="main" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983734 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983738 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="06197912-3b3e-40aa-a19b-8025350bf804" containerName="payload-processing" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983745 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bb011d-a74e-4d78-8a04-79f7eb1dff8f" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983751 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="95df7923-e428-495b-a3a4-ea99cb1b013e" containerName="revoke-keys" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983757 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a79e0dc-3d9b-4309-af30-92c2d76e8c41" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983763 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="842a6154-7c20-43e2-bc07-03a6cce2d71b" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983768 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d31628-1d5e-4601-9b3f-3b514fed875e" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983773 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be90f51d-dc8e-4d0b-b6d4-848c71d2c6ba" containerName="istio-proxy" Oct 31 23:21:34.984320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.983780 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3337f8-211c-4566-a0bd-521baacecd83" containerName="revoke-keys" Oct 31 23:21:34.988872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.988854 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:34.998922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:34.998898 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ba9088\"" Oct 31 23:21:35.005950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.005929 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl"] Oct 31 23:21:35.011124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.011093 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.011255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.011235 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2c8m\" (UniqueName: \"kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m\") pod \"payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.015128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.015098 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" podUID="f23336de-2435-41dd-a86e-629d91536e27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:35.016748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.016683 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:21:35.047628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.047593 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl"] Oct 31 23:21:35.048802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.048782 5043 cpu_manager.go:414] "containerMap: removing container" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" Oct 31 23:21:35.048866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.048803 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" Oct 31 23:21:35.048866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.048864 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" Oct 31 23:21:35.048928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.048870 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1327691f-6ea3-43b8-8b4b-9e38673bbb03" containerName="istio-proxy" Oct 31 23:21:35.048981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.048972 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="651744fd-77a3-431f-8f0e-a2f8509203f5" containerName="istio-proxy" Oct 31 23:21:35.049017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.048986 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b4dacb-d2b9-4af2-8a5c-cd731ef023a4" containerName="istio-proxy" Oct 31 23:21:35.053426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.053399 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.090795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.081310 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ba9088-dockercfg-mmn62\"" Oct 31 23:21:35.216581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.117449 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.216581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.117558 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2c8m\" (UniqueName: \"kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m\") pod \"payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.216581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.136668 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:35.217238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.217160 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.240600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.239394 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2c8m\" (UniqueName: \"kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m\") pod \"payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.252815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.242036 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bp9x\" (UniqueName: \"kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x\") pod \"payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.252815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.242199 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.252815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.247839 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl"] Oct 31 23:21:35.257465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.257384 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng"] Oct 31 23:21:35.262431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.262406 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" event={"ID":"f23336de-2435-41dd-a86e-629d91536e27","Type":"ContainerDied","Data":"e41ab56848321e54fa68b4fb0fec6d4cffdadffe0788b17b291ad551080af372"} Oct 31 23:21:35.262601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.262453 5043 scope.go:122] "RemoveContainer" containerID="a56c04ce6a42b4c3ab48ad4ecb4243c63177ccdfb0d809e8f43ae2151b95cbd1" Oct 31 23:21:35.262675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.262663 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls" Oct 31 23:21:35.276210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.272678 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" event={"ID":"d9bbad83-e55f-4905-941c-cc28506303ed","Type":"ContainerDied","Data":"0eb7bab1c542d7de5e1196b893191f268b577597b154f6bbacc852b0ce8f6526"} Oct 31 23:21:35.276210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.272771 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt" Oct 31 23:21:35.276210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.273781 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac0c695-qgbng"] Oct 31 23:21:35.277497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.277477 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" Oct 31 23:21:35.278883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.278861 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn" event={"ID":"5a0ded48-b028-4b3b-99ac-665ba58e14f6","Type":"ContainerDied","Data":"3a0307ba1cc6d4d6776ab31dc6519c3762d505471cad4f5a90c21b6e126c5c84"} Oct 31 23:21:35.285234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.282464 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" Oct 31 23:21:35.287032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.287011 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7" event={"ID":"3ac6cb61-cfe6-4efc-82db-95b60b80b33a","Type":"ContainerDied","Data":"cf60d5e4efad889555d93ae1e6b8938191c59d853f6aff278bec48a557edd13f"} Oct 31 23:21:35.306001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.305976 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:35.330568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.330517 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls"] Oct 31 23:21:35.346503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.344947 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bp9x\" (UniqueName: \"kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x\") pod \"payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.346503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.345115 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.346503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.346129 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.349142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.349121 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/premium-w5-2294a7-kserve-6d7b7f4f94-8jbls"] Oct 31 23:21:35.353960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.353859 5043 scope.go:122] "RemoveContainer" containerID="897efee68277d18102448bf405190583a7482170251dcd805e74f056d02d63a0" Oct 31 23:21:35.362967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.362944 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bp9x\" (UniqueName: \"kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x\") pod \"payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.404846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.404698 5043 scope.go:122] "RemoveContainer" containerID="2c9638959b2e675c1e2d903c50f23d0278240b35e1e1e31c7331c16adcef7f9e" Oct 31 23:21:35.417737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.417680 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt"] Oct 31 23:21:35.427204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.427050 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-2-w5-2294a7-kserve-7b494d55fd-vcbmt"] Oct 31 23:21:35.434930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.428721 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:21:35.434930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.431055 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ss2sd" Oct 31 23:21:35.434930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.432797 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64f549b4b7-q5w9p" Oct 31 23:21:35.434930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.432896 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ktbm2" Oct 31 23:21:35.484118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.451289 5043 scope.go:122] "RemoveContainer" containerID="4cdb91259c2612b02b50e600a37977a9094e8d130a9fae523b0b2a441a94883b" Oct 31 23:21:35.484118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.476386 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn"] Oct 31 23:21:35.485108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.484838 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:35.494405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.486341 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq"] Oct 31 23:21:35.498958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.498812 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.504358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.504293 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/simulator-w5-2294a7-kserve-58c5569cf9-mrgbn"] Oct 31 23:21:35.510231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.510205 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4eff5f-serving-cert\"" Oct 31 23:21:35.510635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.510479 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:21:35.510985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.510536 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:21:35.511917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.510894 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 23:21:35.511917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.510950 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7j72\"" Oct 31 23:21:35.512831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.512811 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq"] Oct 31 23:21:35.523778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.523503 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7"] Oct 31 23:21:35.533362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.531898 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2294a7/distinct-w5-2294a7-kserve-cc545c8fc-fdck7"] Oct 31 23:21:35.566521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.566476 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.566521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.566526 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.566803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.566633 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45g2m\" (UniqueName: \"kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.566803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.566693 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.583008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.582967 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:35.658851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.658820 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl"] Oct 31 23:21:35.659709 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:35.659670 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bf92b6_7040_4853_810a_87a4ce36f91d.slice/crio-bc6ede944a6852b154390ee3e099b076cc201d8ab051f7cd1600e932adb643e2 WatchSource:0}: Error finding container bc6ede944a6852b154390ee3e099b076cc201d8ab051f7cd1600e932adb643e2: Status 404 returned error can't find the container with id bc6ede944a6852b154390ee3e099b076cc201d8ab051f7cd1600e932adb643e2 Oct 31 23:21:35.668873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.668831 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.668978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.668889 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.668978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.668952 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45g2m\" (UniqueName: \"kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.669100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.669013 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.669251 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:35.669224 5043 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-serving-cert: secret "maas-api-e2e-api-a-4eff5f-serving-cert" not found Oct 31 23:21:35.669333 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:35.669284 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls podName:b8668771-c2e1-46e5-b3c2-d5fb0f57a91e nodeName:}" failed. No retries permitted until 2026-10-31 23:21:36.169265102 +0000 UTC m=+1922.398819504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls") pod "maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" (UID: "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e") : secret "maas-api-e2e-api-a-4eff5f-serving-cert" not found Oct 31 23:21:35.669673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.669647 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.672457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.672427 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.695513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.695457 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45g2m\" (UniqueName: \"kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:35.773096 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.773020 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:35.883923 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:35.883890 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d26d9e9_c130_429e_aa1d_234248fa63aa.slice/crio-c4b212f0dad785cc1e99623b421e6c8f4536d802972773ae938fc0de6b7e98cc WatchSource:0}: Error finding container c4b212f0dad785cc1e99623b421e6c8f4536d802972773ae938fc0de6b7e98cc: Status 404 returned error can't find the container with id c4b212f0dad785cc1e99623b421e6c8f4536d802972773ae938fc0de6b7e98cc Oct 31 23:21:35.884882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.884839 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl"] Oct 31 23:21:35.939519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.939491 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac6cb61-cfe6-4efc-82db-95b60b80b33a" path="/var/lib/kubelet/pods/3ac6cb61-cfe6-4efc-82db-95b60b80b33a/volumes" Oct 31 23:21:35.940130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.940114 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0ded48-b028-4b3b-99ac-665ba58e14f6" path="/var/lib/kubelet/pods/5a0ded48-b028-4b3b-99ac-665ba58e14f6/volumes" Oct 31 23:21:35.940700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.940680 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bbad83-e55f-4905-941c-cc28506303ed" path="/var/lib/kubelet/pods/d9bbad83-e55f-4905-941c-cc28506303ed/volumes" Oct 31 23:21:35.941999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.941980 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e" path="/var/lib/kubelet/pods/f0146c78-6dd7-4ef2-9bd8-f16cc4a6046e/volumes" Oct 31 23:21:35.942473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:35.942458 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23336de-2435-41dd-a86e-629d91536e27" path="/var/lib/kubelet/pods/f23336de-2435-41dd-a86e-629d91536e27/volumes" Oct 31 23:21:36.154040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.153965 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:36.177745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.177711 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:36.180122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.180101 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\") pod \"maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:36.295968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.295935 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" event={"ID":"79bf92b6-7040-4853-810a-87a4ce36f91d","Type":"ContainerStarted","Data":"069b7c2398cd644886f7b4bc89af66b2b207b4d64b4cf992ca64a1c468885243"} Oct 31 23:21:36.296163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.296016 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" event={"ID":"79bf92b6-7040-4853-810a-87a4ce36f91d","Type":"ContainerStarted","Data":"bc6ede944a6852b154390ee3e099b076cc201d8ab051f7cd1600e932adb643e2"} Oct 31 23:21:36.296163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.296043 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:36.300027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.299991 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" event={"ID":"0d26d9e9-c130-429e-aa1d-234248fa63aa","Type":"ContainerStarted","Data":"feb06e08cb89c32e3daba35a09b2b12c8353632ad28f7289d0089d3da099bdc8"} Oct 31 23:21:36.300160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.300050 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" event={"ID":"0d26d9e9-c130-429e-aa1d-234248fa63aa","Type":"ContainerStarted","Data":"c4b212f0dad785cc1e99623b421e6c8f4536d802972773ae938fc0de6b7e98cc"} Oct 31 23:21:36.300160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.300131 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:36.335224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.335175 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" podStartSLOduration=2.335161006 podStartE2EDuration="2.335161006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:36.32925948 +0000 UTC m=+1922.558813889" watchObservedRunningTime="2026-10-31 23:21:36.335161006 +0000 UTC m=+1922.564715413" Oct 31 23:21:36.354938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.354887 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" podStartSLOduration=1.354867252 podStartE2EDuration="1.354867252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:36.351167327 +0000 UTC m=+1922.580721738" watchObservedRunningTime="2026-10-31 23:21:36.354867252 +0000 UTC m=+1922.584421664" Oct 31 23:21:36.439256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.439182 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:36.791598 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:36.787914 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8668771_c2e1_46e5_b3c2_d5fb0f57a91e.slice/crio-4f779a711d0c12e97b4424a81694a61b8774eef063d2a1d0e85a6904a6097a73 WatchSource:0}: Error finding container 4f779a711d0c12e97b4424a81694a61b8774eef063d2a1d0e85a6904a6097a73: Status 404 returned error can't find the container with id 4f779a711d0c12e97b4424a81694a61b8774eef063d2a1d0e85a6904a6097a73 Oct 31 23:21:36.791598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.788700 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq"] Oct 31 23:21:36.920882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:36.920843 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:37.310006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:37.309971 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" event={"ID":"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e","Type":"ContainerStarted","Data":"4f779a711d0c12e97b4424a81694a61b8774eef063d2a1d0e85a6904a6097a73"} Oct 31 23:21:38.047339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:38.047223 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:39.066354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:39.066322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:39.946281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:39.946249 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c5c6d88b7-ftgld" Oct 31 23:21:40.068712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:40.062135 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:40.346379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:40.346325 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" event={"ID":"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e","Type":"ContainerStarted","Data":"4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99"} Oct 31 23:21:40.346572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:40.346426 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:40.380019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:40.379968 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" podStartSLOduration=2.78343285 podStartE2EDuration="5.379954559s" totalImagesPullingTime="2.596521709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:40.378156973 +0000 UTC m=+1926.607711384" watchObservedRunningTime="2026-10-31 23:21:40.379954559 +0000 UTC m=+1926.609508969" Oct 31 23:21:40.460421 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:40.460382 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:21:40.950521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:40.950491 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:21:41.131618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:41.131227 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:41.974433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:41.974401 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:21:41.975016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:41.974989 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:21:41.975746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:41.975728 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:21:42.065561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:42.065521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:42.844121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:42.844079 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7"] Oct 31 23:21:42.852341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:42.852313 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:42.872518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:42.870607 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-885fdf-serving-cert\"" Oct 31 23:21:42.872518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:42.870808 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7"] Oct 31 23:21:43.008221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.008175 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668r8\" (UniqueName: \"kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.008481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.008451 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.008662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.008644 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.008721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.008706 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.088521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.088478 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:43.110352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.110252 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668r8\" (UniqueName: \"kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.110526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.110447 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.111137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.110584 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.111137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.110685 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.111703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.111675 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.118560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.118520 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.118793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.118766 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.135236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.135189 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668r8\" (UniqueName: \"kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8\") pod \"maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.186220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.185852 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:43.264858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.264823 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:43.508714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:43.508682 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7"] Oct 31 23:21:44.244175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.244121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:44.352810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.352490 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g"] Oct 31 23:21:44.361772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.361693 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq"] Oct 31 23:21:44.397187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.397152 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" event={"ID":"00a8347c-472c-4b00-984d-e7b4739b6faa","Type":"ContainerStarted","Data":"d48f1afae4c7c4e5d02315dcf4c3103506cade7a97737749f0119cf6c6c9ab1c"} Oct 31 23:21:44.397336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.397215 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" event={"ID":"00a8347c-472c-4b00-984d-e7b4739b6faa","Type":"ContainerStarted","Data":"f556d7df3be3479deb0259644fddfc43762d42cb1da3c6f67e3f6d49edb45fc3"} Oct 31 23:21:44.397336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.397246 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:44.426963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.426916 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" podStartSLOduration=2.010247865 podStartE2EDuration="2.426902474s" totalImagesPullingTime="416.654609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:44.426255156 +0000 UTC m=+1930.655809566" watchObservedRunningTime="2026-10-31 23:21:44.426902474 +0000 UTC m=+1930.656456883" Oct 31 23:21:44.443606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.443573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:44.643032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:44.642898 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:45.370414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:45.370362 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:45.406416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:45.406379 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4vgvf" Oct 31 23:21:46.144847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:46.144772 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:46.365616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:46.365582 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:21:46.987839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:46.987734 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:47.313305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:47.313278 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:21:47.313524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:47.313359 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:21:47.851020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:47.850700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:48.004644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:48.004612 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:48.729298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:48.729265 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:48.889832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:48.889797 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:49.050745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.050712 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:49.353840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.353747 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" containerID="cri-o://7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2" gracePeriod=25 Oct 31 23:21:49.362597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.362534 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" containerID="cri-o://d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b" gracePeriod=25 Oct 31 23:21:49.687992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.687962 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:21:49.744443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.744391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:49.885093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.885051 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume\") pod \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " Oct 31 23:21:49.885298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.885257 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6\" (UniqueName: \"kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6\") pod \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\" (UID: \"d046db04-b4c5-4213-9bbf-5b6728d7a6f1\") " Oct 31 23:21:49.885502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.885475 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume" pod "d046db04-b4c5-4213-9bbf-5b6728d7a6f1" (UID: "d046db04-b4c5-4213-9bbf-5b6728d7a6f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:21:49.886102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.886079 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:49.887611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.887579 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6" pod "d046db04-b4c5-4213-9bbf-5b6728d7a6f1" (UID: "d046db04-b4c5-4213-9bbf-5b6728d7a6f1"). InnerVolumeSpecName "kube-api-access-2xvg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:21:49.925013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.924983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:49.988118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:49.988086 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xvg6\" (UniqueName: \"kubernetes.io/projected/d046db04-b4c5-4213-9bbf-5b6728d7a6f1-kube-api-access-2xvg6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:21:50.351565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.334189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:50.418275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.418246 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:21:50.481386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.481349 5043 generic.go:356] "Generic (PLEG): container finished" podID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerID="d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b" exitCode=0 Oct 31 23:21:50.481720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.481406 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" Oct 31 23:21:50.481848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.481430 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" event={"ID":"d046db04-b4c5-4213-9bbf-5b6728d7a6f1","Type":"ContainerDied","Data":"d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b"} Oct 31 23:21:50.481920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.481848 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq" event={"ID":"d046db04-b4c5-4213-9bbf-5b6728d7a6f1","Type":"ContainerDied","Data":"9a167ab750fc1d390bd452e6ea204d5f37db724ec6f7c945776de2ee5bcab1b9"} Oct 31 23:21:50.481920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.481884 5043 scope.go:122] "RemoveContainer" containerID="d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b" Oct 31 23:21:50.489945 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.489912 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:50.498873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.498855 5043 scope.go:122] "RemoveContainer" containerID="d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b" Oct 31 23:21:50.499167 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:21:50.499141 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b\": container with ID starting with d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b not found: ID does not exist" containerID="d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b" Oct 31 23:21:50.499265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.499179 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b"} err="failed to get container status \"d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b\": rpc error: code = NotFound desc = could not find container \"d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b\": container with ID starting with d8abca04bd779a70c8657c1f84170027dd3b2f2f14edf22a434a693c3aa3542b not found: ID does not exist" Oct 31 23:21:50.510438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.510415 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq"] Oct 31 23:21:50.531855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.531829 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-36a15fc5-fdc4954d-qz9bq"] Oct 31 23:21:50.653628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.653281 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:50.840123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.840091 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:50.988799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:50.988718 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:51.131933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.131888 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:51.492441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.492411 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:51.694567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.694078 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv"] Oct 31 23:21:51.743618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.742973 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:51.777856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.777812 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" Oct 31 23:21:51.777856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.777848 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" Oct 31 23:21:51.778196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.778174 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" containerName="payload-processing" Oct 31 23:21:51.796804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.787630 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.807584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.800517 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv"] Oct 31 23:21:51.864163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.863737 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a1deb6-openshift-default-dockercfg-sx6st\"" Oct 31 23:21:51.874644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.874611 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875111 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875168 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875202 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875282 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875329 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875399 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmdc\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875424 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.875603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.875470 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.941416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.941380 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d046db04-b4c5-4213-9bbf-5b6728d7a6f1" path="/var/lib/kubelet/pods/d046db04-b4c5-4213-9bbf-5b6728d7a6f1/volumes" Oct 31 23:21:51.977023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.976995 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977029 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977104 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977149 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977216 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmdc\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977244 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977292 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977434 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.977657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977638 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.978475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.977992 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.978475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.978078 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.978475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.978214 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.978475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.978430 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.979979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.979952 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.980957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.980938 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.992854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.992822 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmdc\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:51.992984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:51.992968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:52.003249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.003177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:52.116841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.116800 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:52.426572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.426445 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:52.450952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.449222 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv"] Oct 31 23:21:52.451078 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:52.451038 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe93bc4_f2d7_4543_a75a_da1bb9e18cb0.slice/crio-1f90712791ccc763bc2db1a070f48d307a008450df9dfd22e6b894ebee72a73d WatchSource:0}: Error finding container 1f90712791ccc763bc2db1a070f48d307a008450df9dfd22e6b894ebee72a73d: Status 404 returned error can't find the container with id 1f90712791ccc763bc2db1a070f48d307a008450df9dfd22e6b894ebee72a73d Oct 31 23:21:52.462575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.455963 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:52.462575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.456054 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:52.462575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.456096 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:21:52.503194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:52.503168 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" event={"ID":"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0","Type":"ContainerStarted","Data":"1f90712791ccc763bc2db1a070f48d307a008450df9dfd22e6b894ebee72a73d"} Oct 31 23:21:53.277234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:53.277200 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:53.516436 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:53.516400 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" event={"ID":"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0","Type":"ContainerStarted","Data":"6b83c0f880f764c89884bd5c57afb6261257f1ebdc75e46acd9c162938987327"} Oct 31 23:21:54.023205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.020131 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podStartSLOduration=3.020110586 podStartE2EDuration="3.020110586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:53.561483061 +0000 UTC m=+1939.791037470" watchObservedRunningTime="2026-10-31 23:21:54.020110586 +0000 UTC m=+1940.249664996" Oct 31 23:21:54.042567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.037600 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:21:54.052220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.052191 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9"] Oct 31 23:21:54.065571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.062985 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9"] Oct 31 23:21:54.065571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.063084 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.075218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.071477 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ba9088-serving-cert\"" Oct 31 23:21:54.104947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.104918 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:54.132775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.117696 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:54.132775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.127328 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc9q\" (UniqueName: \"kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.132775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.127444 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.132775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.127681 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.132775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.127851 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.132775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.132093 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:54.229533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.229500 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.229772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.229659 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc9q\" (UniqueName: \"kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.229772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.229714 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.229844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.229796 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.230358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.230334 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.232401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.232375 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.232596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.232534 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.243591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.243556 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc9q\" (UniqueName: \"kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q\") pod \"maas-api-e2e-api-b-ba9088-7469f567cd-f29x9\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.253849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.253812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:54.388924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.388835 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:54.419927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.419885 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:54.536639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.536608 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:54.538297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.538277 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:21:54.778231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.777754 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9"] Oct 31 23:21:54.945556 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:54.945509 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:55.169348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:55.169263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:55.566891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:55.566854 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" event={"ID":"5f9cb063-6cf6-4461-87af-5d1e48affa27","Type":"ContainerStarted","Data":"2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8"} Oct 31 23:21:55.567380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:55.567090 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:21:55.567380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:55.567174 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" event={"ID":"5f9cb063-6cf6-4461-87af-5d1e48affa27","Type":"ContainerStarted","Data":"b8d64b97d8c5bd1f94d85c36c123c423acc85de30380625d186677cd96eb1b39"} Oct 31 23:21:55.606906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:55.606863 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" podStartSLOduration=2.144114606 podStartE2EDuration="2.606851197s" totalImagesPullingTime="462.736591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:55.602086616 +0000 UTC m=+1941.831641027" watchObservedRunningTime="2026-10-31 23:21:55.606851197 +0000 UTC m=+1941.836405606" Oct 31 23:21:56.013995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:56.013956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:56.252024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:56.251984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:56.479115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:56.479033 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:57.160704 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.159387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:57.509909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.509876 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:57.510576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.510538 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw"] Oct 31 23:21:57.529744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.529697 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c"] Oct 31 23:21:57.530110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.530089 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:57.536525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.536497 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a1deb6\"" Oct 31 23:21:57.539195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.538044 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a1deb6-dockercfg-4s7q4\"" Oct 31 23:21:57.551933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.547771 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw"] Oct 31 23:21:57.551933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.547808 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c"] Oct 31 23:21:57.551933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.547895 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.682592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.678685 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:57.682592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.681417 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.682592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.681482 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqh8\" (UniqueName: \"kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8\") pod \"payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.788204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.788107 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbt6\" (UniqueName: \"kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6\") pod \"payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:57.788479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.788459 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:57.788757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.788735 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.793636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.793609 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqh8\" (UniqueName: \"kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8\") pod \"payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.809590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.795306 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.855734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.855700 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqh8\" (UniqueName: \"kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8\") pod \"payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.867576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.866796 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:57.896568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.895317 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbt6\" (UniqueName: \"kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6\") pod \"payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:57.896568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.895422 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:57.898598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.898280 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:57.917373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:57.917345 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbt6\" (UniqueName: \"kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6\") pod \"payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:58.169444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.168968 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:58.371251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.371211 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c"] Oct 31 23:21:58.606622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.606582 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" event={"ID":"aa780837-0bd4-4676-88be-94ebb229cf22","Type":"ContainerStarted","Data":"40a8669644bc1096b91076162178f825184dc194d93146e6c17a306b8c269de8"} Oct 31 23:21:58.606823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.606655 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" event={"ID":"aa780837-0bd4-4676-88be-94ebb229cf22","Type":"ContainerStarted","Data":"7eb44700246657c11650991c06b14aae1bea099d30f92fb79d3f191e2b482ca0"} Oct 31 23:21:58.606876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.606843 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:21:58.632769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.632695 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" podStartSLOduration=1.632670821 podStartE2EDuration="1.632670821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:58.629184328 +0000 UTC m=+1944.858738740" watchObservedRunningTime="2026-10-31 23:21:58.632670821 +0000 UTC m=+1944.862225230" Oct 31 23:21:58.641115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.641080 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:58.687530 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:21:58.687461 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a540c7_f029_4183_80f3_997c9cf3e3a7.slice/crio-0677514eb0c5ae19b4c76e530759cfee54aa397f4750d2875c9823908c61bc01 WatchSource:0}: Error finding container 0677514eb0c5ae19b4c76e530759cfee54aa397f4750d2875c9823908c61bc01: Status 404 returned error can't find the container with id 0677514eb0c5ae19b4c76e530759cfee54aa397f4750d2875c9823908c61bc01 Oct 31 23:21:58.689037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.689012 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw"] Oct 31 23:21:58.809296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:58.809264 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:59.290814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:59.290762 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:59.577623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:59.575215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:21:59.632570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:59.632519 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" event={"ID":"f1a540c7-f029-4183-80f3-997c9cf3e3a7","Type":"ContainerStarted","Data":"25b9be2fd3fdc3e9cffb55ee4182708450237f5e899d92260e1788270052d106"} Oct 31 23:21:59.632747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:59.632634 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" event={"ID":"f1a540c7-f029-4183-80f3-997c9cf3e3a7","Type":"ContainerStarted","Data":"0677514eb0c5ae19b4c76e530759cfee54aa397f4750d2875c9823908c61bc01"} Oct 31 23:21:59.632747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:59.632688 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:21:59.952316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:21:59.952228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:00.330217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.329329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:00.336939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.336885 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" podStartSLOduration=3.336867672 podStartE2EDuration="3.336867672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:21:59.662768556 +0000 UTC m=+1945.892322965" watchObservedRunningTime="2026-10-31 23:22:00.336867672 +0000 UTC m=+1946.566422087" Oct 31 23:22:00.345055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.344902 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h"] Oct 31 23:22:00.371113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.371077 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h"] Oct 31 23:22:00.371298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.371193 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.378010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.377392 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-836c99-openshift-default-dockercfg-zn6fh\"" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.474635 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.474747 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.474790 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.474842 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.474981 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.475028 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.475091 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.475245 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.479602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.475285 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.521583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.518394 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.576979 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577132 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577170 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577224 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577353 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577491 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577567 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.577603 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.578043 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.578797 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.579010 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.581059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.579220 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.595607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.594127 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.595607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.594376 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.612247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.605588 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.612247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.609335 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.632130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.615111 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.696845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.696809 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:00.718567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:00.716335 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:01.042515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.042469 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h"] Oct 31 23:22:01.050528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.050488 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:01.050802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.050597 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:01.050802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.050658 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:01.366030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.365944 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:01.582202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.582167 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:22:01.594676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.594632 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:01.703979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.703885 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" event={"ID":"a302ec33-70dd-42ae-baa1-aaa48d20a369","Type":"ContainerStarted","Data":"ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50"} Oct 31 23:22:01.703979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.703955 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" event={"ID":"a302ec33-70dd-42ae-baa1-aaa48d20a369","Type":"ContainerStarted","Data":"33b73166b543624a7037bba9edfa3eba70da968a9b240a451f91ac96ef87b2cb"} Oct 31 23:22:01.747284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.747201 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podStartSLOduration=1.7471820569999998 podStartE2EDuration="1.747182057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:01.741223 +0000 UTC m=+1947.970777411" watchObservedRunningTime="2026-10-31 23:22:01.747182057 +0000 UTC m=+1947.976736468" Oct 31 23:22:01.762845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:01.762807 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:02.030522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.030388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:02.429023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.428913 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s"] Oct 31 23:22:02.441410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.441379 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.443799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.443765 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s"] Oct 31 23:22:02.449820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.449796 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:22:02.449976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.449891 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:22:02.540852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.532424 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchnz\" (UniqueName: \"kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.540852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.532642 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.540852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.532697 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.557609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.547671 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:02.634214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.634175 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.634214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.634239 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.634534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.634513 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchnz\" (UniqueName: \"kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.635298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.635280 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.657040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.653725 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.663586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.663339 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchnz\" (UniqueName: \"kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz\") pod \"maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:02.697978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.697898 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:02.708895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.708848 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:02.719592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.718077 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:02.719592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.718965 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:22:02.865561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:02.865502 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:03.602567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:03.572902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:03.727653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:03.716634 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s"] Oct 31 23:22:03.867850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:03.867754 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" event={"ID":"47687c36-f155-4842-b818-5f5133494786","Type":"ContainerStarted","Data":"993eedb5dd1599bc31038387d3b2ca1fde1c671bfc09936c17d014b578fd3fa9"} Oct 31 23:22:04.088579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:04.085820 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:22:04.096902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:04.096850 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:22:04.659226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:04.656131 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:04.660438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:04.660369 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:04.823031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:04.822992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:05.018475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.018089 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:05.048573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.037728 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" event={"ID":"47687c36-f155-4842-b818-5f5133494786","Type":"ContainerStarted","Data":"342ece31fb09639319bc0223d0d53d72e365580301a498f53b963b45c9a67227"} Oct 31 23:22:05.125625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.100449 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:22:05.125625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.101045 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:22:05.205593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.174947 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" podStartSLOduration=3.174924692 podStartE2EDuration="3.174924692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:05.074781331 +0000 UTC m=+1951.304335743" watchObservedRunningTime="2026-10-31 23:22:05.174924692 +0000 UTC m=+1951.404479102" Oct 31 23:22:05.723572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.716888 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:05.723572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.717250 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:22:05.883571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:05.883517 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:22:06.027569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.014921 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:06.122172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.100611 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:06.330383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.329637 5043 generic.go:356] "Generic (PLEG): container finished" podID="47687c36-f155-4842-b818-5f5133494786" containerID="342ece31fb09639319bc0223d0d53d72e365580301a498f53b963b45c9a67227" exitCode=0 Oct 31 23:22:06.410144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.410048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:06.603624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.602925 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" event={"ID":"47687c36-f155-4842-b818-5f5133494786","Type":"ContainerDied","Data":"342ece31fb09639319bc0223d0d53d72e365580301a498f53b963b45c9a67227"} Oct 31 23:22:06.603624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.603031 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:22:06.617956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.608649 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799"] Oct 31 23:22:06.634971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.633648 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:06.650211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.650183 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:06.727812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.727766 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-836c99\"" Oct 31 23:22:06.729610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.729583 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799"] Oct 31 23:22:06.803575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.752638 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t"] Oct 31 23:22:06.803575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.779769 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:06.803575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.779933 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjth\" (UniqueName: \"kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth\") pod \"payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.888562 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.888735 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjth\" (UniqueName: \"kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth\") pod \"payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.891814 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t"] Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.891947 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.896147 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.896214 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:22:06.924793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.904428 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-836c99-dockercfg-94dfj\"" Oct 31 23:22:06.993580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:06.993322 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjth\" (UniqueName: \"kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth\") pod \"payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:07.019227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.017716 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume\") pod \"payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.019227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.017801 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9b2\" (UniqueName: \"kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2\") pod \"payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.127571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.126933 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume\") pod \"payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.127571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.126997 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9b2\" (UniqueName: \"kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2\") pod \"payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.153570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.144350 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume\") pod \"payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.193614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.191683 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9b2\" (UniqueName: \"kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2\") pod \"payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.203278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.199962 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:07.319595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.318300 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:07.876344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.875946 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:07.934346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.934284 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:07.980338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.980247 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca\") pod \"47687c36-f155-4842-b818-5f5133494786\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " Oct 31 23:22:07.981228 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.981205 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca\") pod \"47687c36-f155-4842-b818-5f5133494786\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " Oct 31 23:22:07.982050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.982025 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz\" (UniqueName: \"kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz\") pod \"47687c36-f155-4842-b818-5f5133494786\" (UID: \"47687c36-f155-4842-b818-5f5133494786\") " Oct 31 23:22:07.983312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.981702 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca" pod "47687c36-f155-4842-b818-5f5133494786" (UID: "47687c36-f155-4842-b818-5f5133494786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:08.019671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:07.995850 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz" pod "47687c36-f155-4842-b818-5f5133494786" (UID: "47687c36-f155-4842-b818-5f5133494786"). InnerVolumeSpecName "kube-api-access-lchnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:08.019671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.017096 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca" pod "47687c36-f155-4842-b818-5f5133494786" (UID: "47687c36-f155-4842-b818-5f5133494786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:08.074283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.074252 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799"] Oct 31 23:22:08.087587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.084705 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47687c36-f155-4842-b818-5f5133494786-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:08.087587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.084737 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchnz\" (UniqueName: \"kubernetes.io/projected/47687c36-f155-4842-b818-5f5133494786-kube-api-access-lchnz\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:08.087587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.084756 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47687c36-f155-4842-b818-5f5133494786-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:08.143499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.143457 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t"] Oct 31 23:22:08.381616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.381574 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" event={"ID":"58194d01-fc01-4f35-81ca-b242916ded14","Type":"ContainerStarted","Data":"1b2b71a46463d0e8d384c6c41718dd359fc93eeb14201783726e13eede66e287"} Oct 31 23:22:08.381819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.381649 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:08.381819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.381676 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" event={"ID":"58194d01-fc01-4f35-81ca-b242916ded14","Type":"ContainerStarted","Data":"59c8fb40201f2905179c3ac9b7517482c1dd48443a705ed4234106a7012fffea"} Oct 31 23:22:08.383462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.383432 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" event={"ID":"bddb658d-2e37-46b8-a0e4-92d2c28f7039","Type":"ContainerStarted","Data":"5c225d4d5b483d057cfab751c86890ec44759bbc78ed338ead6c4639751d704a"} Oct 31 23:22:08.383634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.383483 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" event={"ID":"bddb658d-2e37-46b8-a0e4-92d2c28f7039","Type":"ContainerStarted","Data":"874d432523443ad5821aa9bea61a7f9d281d3fb457df15b10dc8b73004e2e0a2"} Oct 31 23:22:08.383634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.383515 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:08.385110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.385087 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" event={"ID":"47687c36-f155-4842-b818-5f5133494786","Type":"ContainerDied","Data":"993eedb5dd1599bc31038387d3b2ca1fde1c671bfc09936c17d014b578fd3fa9"} Oct 31 23:22:08.385110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.385099 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s" Oct 31 23:22:08.385303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.385121 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993eedb5dd1599bc31038387d3b2ca1fde1c671bfc09936c17d014b578fd3fa9" Oct 31 23:22:08.406219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.406163 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" podStartSLOduration=2.40614542 podStartE2EDuration="2.40614542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:08.404330137 +0000 UTC m=+1954.633884547" watchObservedRunningTime="2026-10-31 23:22:08.40614542 +0000 UTC m=+1954.635699831" Oct 31 23:22:08.432811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:08.432757 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" podStartSLOduration=2.432744016 podStartE2EDuration="2.432744016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:08.431762024 +0000 UTC m=+1954.661316434" watchObservedRunningTime="2026-10-31 23:22:08.432744016 +0000 UTC m=+1954.662298482" Oct 31 23:22:09.263527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.263488 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:09.657806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.653129 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:22:09.694051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.694023 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:22:09.847529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.847433 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478\" (UniqueName: \"kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478\") pod \"263eeccd-41ea-40ad-aa08-d8470d5885d4\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " Oct 31 23:22:09.847743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.847687 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume\") pod \"263eeccd-41ea-40ad-aa08-d8470d5885d4\" (UID: \"263eeccd-41ea-40ad-aa08-d8470d5885d4\") " Oct 31 23:22:09.848076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.848046 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume" pod "263eeccd-41ea-40ad-aa08-d8470d5885d4" (UID: "263eeccd-41ea-40ad-aa08-d8470d5885d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:09.848490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.848468 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263eeccd-41ea-40ad-aa08-d8470d5885d4-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:09.849848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.849818 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478" pod "263eeccd-41ea-40ad-aa08-d8470d5885d4" (UID: "263eeccd-41ea-40ad-aa08-d8470d5885d4"). InnerVolumeSpecName "kube-api-access-qk478". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:09.950449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:09.950411 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk478\" (UniqueName: \"kubernetes.io/projected/263eeccd-41ea-40ad-aa08-d8470d5885d4-kube-api-access-qk478\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:10.181387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.181288 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:10.406631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.406600 5043 generic.go:356] "Generic (PLEG): container finished" podID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerID="7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2" exitCode=0 Oct 31 23:22:10.407119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.406652 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" Oct 31 23:22:10.407119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.406687 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" event={"ID":"263eeccd-41ea-40ad-aa08-d8470d5885d4","Type":"ContainerDied","Data":"7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2"} Oct 31 23:22:10.407119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.406749 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g" event={"ID":"263eeccd-41ea-40ad-aa08-d8470d5885d4","Type":"ContainerDied","Data":"26ceb855e73c4a356e7850cdb0770e4fca34194d7cfc5f1237f6d9c5a80aad60"} Oct 31 23:22:10.407119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.406779 5043 scope.go:122] "RemoveContainer" containerID="7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2" Oct 31 23:22:10.427853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.427325 5043 scope.go:122] "RemoveContainer" containerID="7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2" Oct 31 23:22:10.429239 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:10.429012 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2\": container with ID starting with 7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2 not found: ID does not exist" containerID="7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2" Oct 31 23:22:10.429239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.429068 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2"} err="failed to get container status \"7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2\": rpc error: code = NotFound desc = could not find container \"7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2\": container with ID starting with 7f273186da71b12c580286e974f1e45e26b0bd84ea087b93663a324ba29752a2 not found: ID does not exist" Oct 31 23:22:10.446898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.446827 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g"] Oct 31 23:22:10.452621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.452320 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36a15fc5-65955b56dc-wrv6g"] Oct 31 23:22:10.473119 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:10.473085 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:22:10.632568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.626931 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:10.695322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:10.695289 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:22:11.047997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.047955 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:11.502620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.502584 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp"] Oct 31 23:22:11.503410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.503394 5043 cpu_manager.go:414] "containerMap: removing container" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" Oct 31 23:22:11.503462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.503412 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" Oct 31 23:22:11.503462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.503454 5043 cpu_manager.go:414] "containerMap: removing container" podUID="47687c36-f155-4842-b818-5f5133494786" containerName="revoke-keys" Oct 31 23:22:11.503462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.503460 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47687c36-f155-4842-b818-5f5133494786" containerName="revoke-keys" Oct 31 23:22:11.503656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.503626 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="47687c36-f155-4842-b818-5f5133494786" containerName="revoke-keys" Oct 31 23:22:11.503656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.503657 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" containerName="payload-pre-processing" Oct 31 23:22:11.539644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.522062 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.555057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.544138 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp"] Oct 31 23:22:11.604956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.604924 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:22:11.605114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.605034 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:22:11.679957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.679921 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.680134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.680105 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.680304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.680285 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpdw\" (UniqueName: \"kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.782187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.782105 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpdw\" (UniqueName: \"kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.782366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.782248 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.782366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.782337 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.783087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.783063 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.784781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.784759 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.795021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.794997 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpdw\" (UniqueName: \"kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw\") pod \"maas-api-revoke-keys-e2e-api-a-479107-zlhhp\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.871979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.871722 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:11.968929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:11.968432 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263eeccd-41ea-40ad-aa08-d8470d5885d4" path="/var/lib/kubelet/pods/263eeccd-41ea-40ad-aa08-d8470d5885d4/volumes" Oct 31 23:22:12.185451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:12.185349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:12.204849 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:12.204818 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d19494_8b02_4c36_a383_8383d45a933c.slice/crio-0ee762d8638b4393184157b0c9f5f2296e7ca65ba7ae98308352f6dcb6749713 WatchSource:0}: Error finding container 0ee762d8638b4393184157b0c9f5f2296e7ca65ba7ae98308352f6dcb6749713: Status 404 returned error can't find the container with id 0ee762d8638b4393184157b0c9f5f2296e7ca65ba7ae98308352f6dcb6749713 Oct 31 23:22:12.206806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:12.206778 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp"] Oct 31 23:22:12.444493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:12.444449 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" event={"ID":"65d19494-8b02-4c36-a383-8383d45a933c","Type":"ContainerStarted","Data":"0ee762d8638b4393184157b0c9f5f2296e7ca65ba7ae98308352f6dcb6749713"} Oct 31 23:22:12.786219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:12.786184 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:12.978265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:12.978147 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:13.175173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:13.175087 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:13.392205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:13.392164 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:13.460029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:13.459689 5043 generic.go:356] "Generic (PLEG): container finished" podID="65d19494-8b02-4c36-a383-8383d45a933c" containerID="f1e56a82f231642d94eadd189b600828069e1088432670a668cdbc37a3404225" exitCode=0 Oct 31 23:22:13.460029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:13.459772 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" event={"ID":"65d19494-8b02-4c36-a383-8383d45a933c","Type":"ContainerDied","Data":"f1e56a82f231642d94eadd189b600828069e1088432670a668cdbc37a3404225"} Oct 31 23:22:13.847458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:13.838601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:14.282491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.282456 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:14.535583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.534869 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:14.776213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.776190 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:14.839715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.839632 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw\" (UniqueName: \"kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw\") pod \"65d19494-8b02-4c36-a383-8383d45a933c\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " Oct 31 23:22:14.839891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.839872 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca\") pod \"65d19494-8b02-4c36-a383-8383d45a933c\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " Oct 31 23:22:14.840032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.840006 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca\") pod \"65d19494-8b02-4c36-a383-8383d45a933c\" (UID: \"65d19494-8b02-4c36-a383-8383d45a933c\") " Oct 31 23:22:14.841034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.840362 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca" pod "65d19494-8b02-4c36-a383-8383d45a933c" (UID: "65d19494-8b02-4c36-a383-8383d45a933c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:14.842129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.842102 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw" pod "65d19494-8b02-4c36-a383-8383d45a933c" (UID: "65d19494-8b02-4c36-a383-8383d45a933c"). InnerVolumeSpecName "kube-api-access-5lpdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:14.843387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.843364 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca" pod "65d19494-8b02-4c36-a383-8383d45a933c" (UID: "65d19494-8b02-4c36-a383-8383d45a933c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:14.942883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.942844 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:14.942883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.942876 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lpdw\" (UniqueName: \"kubernetes.io/projected/65d19494-8b02-4c36-a383-8383d45a933c-kube-api-access-5lpdw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:14.942883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:14.942887 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d19494-8b02-4c36-a383-8383d45a933c-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:15.345213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:15.345178 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:15.482397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:15.482360 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" event={"ID":"65d19494-8b02-4c36-a383-8383d45a933c","Type":"ContainerDied","Data":"0ee762d8638b4393184157b0c9f5f2296e7ca65ba7ae98308352f6dcb6749713"} Oct 31 23:22:15.482592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:15.482421 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee762d8638b4393184157b0c9f5f2296e7ca65ba7ae98308352f6dcb6749713" Oct 31 23:22:15.482592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:15.482373 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp" Oct 31 23:22:16.063218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.063163 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:16.803160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.803113 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx"] Oct 31 23:22:16.804525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.804491 5043 cpu_manager.go:414] "containerMap: removing container" podUID="65d19494-8b02-4c36-a383-8383d45a933c" containerName="revoke-keys" Oct 31 23:22:16.804525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.804524 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d19494-8b02-4c36-a383-8383d45a933c" containerName="revoke-keys" Oct 31 23:22:16.804749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.804732 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d19494-8b02-4c36-a383-8383d45a933c" containerName="revoke-keys" Oct 31 23:22:16.814574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.811760 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx"] Oct 31 23:22:16.814574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.811873 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:16.842569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.842226 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a1deb6-serving-cert\"" Oct 31 23:22:16.895926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.886474 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:16.895926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.886799 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:16.895926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.886966 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:16.895926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.887144 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9njg\" (UniqueName: \"kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:16.930571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.925387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.998124 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.998254 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.998477 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9njg\" (UniqueName: \"kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:16.998593 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:16.999351 5043 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-serving-cert: secret "maas-api-e2e-api-a-a1deb6-serving-cert" not found Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:16.999413 5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls podName:150e336e-e9c8-4c1d-87e3-2f1b61a8ca86 nodeName:}" failed. No retries permitted until 2026-10-31 23:22:17.499392971 +0000 UTC m=+1963.728947365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls") pod "maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" (UID: "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86") : secret "maas-api-e2e-api-a-a1deb6-serving-cert" not found Oct 31 23:22:17.009566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.000130 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.047925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.028675 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9njg\" (UniqueName: \"kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.055947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.055611 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.119811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.119777 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:17.294897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.294858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:17.508149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.508116 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.511139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.511105 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\") pod \"maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:17.795666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:17.776748 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:18.095417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:18.095310 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:18.111583 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:18.111510 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150e336e_e9c8_4c1d_87e3_2f1b61a8ca86.slice/crio-0937fe9d38ca23050ad654b7e378de929515517941fe579c6031871fffd938f8 WatchSource:0}: Error finding container 0937fe9d38ca23050ad654b7e378de929515517941fe579c6031871fffd938f8: Status 404 returned error can't find the container with id 0937fe9d38ca23050ad654b7e378de929515517941fe579c6031871fffd938f8 Oct 31 23:22:18.119641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:18.119609 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx"] Oct 31 23:22:18.512752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:18.512719 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" event={"ID":"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86","Type":"ContainerStarted","Data":"0937fe9d38ca23050ad654b7e378de929515517941fe579c6031871fffd938f8"} Oct 31 23:22:19.214907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.214852 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:19.381802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.381767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:19.399143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.399097 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:22:19.400995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.400968 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:22:19.546523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.546483 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" event={"ID":"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86","Type":"ContainerStarted","Data":"fef5f76cf68e6867af3954b0c64e9c5e8d39b99ef1daad3a8fd96f4103f84499"} Oct 31 23:22:19.547029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.546633 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:19.575316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:19.575250 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" podStartSLOduration=3.136123337 podStartE2EDuration="3.575231712s" totalImagesPullingTime="439.108375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:19.573993988 +0000 UTC m=+1965.803548400" watchObservedRunningTime="2026-10-31 23:22:19.575231712 +0000 UTC m=+1965.804786123" Oct 31 23:22:20.271826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.271790 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds"] Oct 31 23:22:20.276933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.276908 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.302639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.302609 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds"] Oct 31 23:22:20.342266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.341778 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4eff5f-openshift-default-dockercfg-lfwft\"" Oct 31 23:22:20.346286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.346215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.365712 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.375715 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.375765 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.375942 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.375990 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.376183 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.376316 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqzh\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.376370 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.378749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.376406 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.477992 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478162 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478210 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478330 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478441 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqzh\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478485 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478514 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478585 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478617 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478668 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.478932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478892 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.479004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478940 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.479004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.478960 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.479107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.479027 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.480771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.480749 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.480986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.480968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.491050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.491029 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.491184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.491166 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqzh\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh\") pod \"e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.594192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.594107 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:20.836603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.836567 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4"] Oct 31 23:22:20.861619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.861476 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:20.863962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.863936 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4"] Oct 31 23:22:20.865991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.865965 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:22:20.866198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.866183 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:22:20.906230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.906200 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds"] Oct 31 23:22:20.908541 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.908511 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:20.908695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.908606 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:20.908695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.908651 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:20.986175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.986150 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:20.986400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.986387 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:20.986515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:20.986493 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrd7\" (UniqueName: \"kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.083259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.083221 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:21.090562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.090515 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.090771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.090722 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrd7\" (UniqueName: \"kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.090850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.090826 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.092563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.091945 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.099194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.096974 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.106105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.106053 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrd7\" (UniqueName: \"kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7\") pod \"maas-api-revoke-sub-9917efc9-qzpn4\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.181231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.181150 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:21.506133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.506099 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4"] Oct 31 23:22:21.507994 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:21.507968 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e494ca7_3dba_4df4_ad21_fcddadc8121d.slice/crio-29e08121d4da921da4e0c3195bd39490eafc8614125f2b6e42e670e5eb869501 WatchSource:0}: Error finding container 29e08121d4da921da4e0c3195bd39490eafc8614125f2b6e42e670e5eb869501: Status 404 returned error can't find the container with id 29e08121d4da921da4e0c3195bd39490eafc8614125f2b6e42e670e5eb869501 Oct 31 23:22:21.570115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.570083 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" event={"ID":"a231fc23-2c9b-4337-bc59-fd95964c2a2f","Type":"ContainerStarted","Data":"7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04"} Oct 31 23:22:21.570269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.570138 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" event={"ID":"a231fc23-2c9b-4337-bc59-fd95964c2a2f","Type":"ContainerStarted","Data":"45cc7139f153004b0bb732d5c5b3ca4e2c2e42f5cce86f6aca29d1e8f83dcc0e"} Oct 31 23:22:21.571625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.571601 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" event={"ID":"0e494ca7-3dba-4df4-ad21-fcddadc8121d","Type":"ContainerStarted","Data":"29e08121d4da921da4e0c3195bd39490eafc8614125f2b6e42e670e5eb869501"} Oct 31 23:22:21.595204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.595175 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:21.596904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.596881 5043 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 23:22:21.597020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.596941 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 23:22:21.602758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.602717 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podStartSLOduration=1.602705427 podStartE2EDuration="1.602705427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:21.598906232 +0000 UTC m=+1967.828460643" watchObservedRunningTime="2026-10-31 23:22:21.602705427 +0000 UTC m=+1967.832259837" Oct 31 23:22:21.771377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.771096 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:21.935574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:21.935517 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:22.273317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:22.273280 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:22.439091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:22.438767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:22.583311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:22.583269 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" event={"ID":"0e494ca7-3dba-4df4-ad21-fcddadc8121d","Type":"ContainerStarted","Data":"f06c6e509606149b7d8405bb0e6d2c2d3d42752513a275afadd1e67d9bd7b181"} Oct 31 23:22:22.594819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:22.594792 5043 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 23:22:22.594972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:22.594846 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 23:22:22.639135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:22.633820 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" podStartSLOduration=2.633800754 podStartE2EDuration="2.633800754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:22.632278584 +0000 UTC m=+1968.861832996" watchObservedRunningTime="2026-10-31 23:22:22.633800754 +0000 UTC m=+1968.863355167" Oct 31 23:22:23.442572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:23.431617 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:23.595016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:23.594984 5043 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 23:22:23.595195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:23.595038 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 23:22:23.596920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:23.596894 5043 generic.go:356] "Generic (PLEG): container finished" podID="0e494ca7-3dba-4df4-ad21-fcddadc8121d" containerID="f06c6e509606149b7d8405bb0e6d2c2d3d42752513a275afadd1e67d9bd7b181" exitCode=0 Oct 31 23:22:23.597067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:23.596924 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" event={"ID":"0e494ca7-3dba-4df4-ad21-fcddadc8121d","Type":"ContainerDied","Data":"f06c6e509606149b7d8405bb0e6d2c2d3d42752513a275afadd1e67d9bd7b181"} Oct 31 23:22:24.175696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:24.174572 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:24.576584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:24.576536 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6"] Oct 31 23:22:24.576830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:24.576807 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" containerID="cri-o://96003ef535034fb9211ba89a8bbf1708a2d9944b47b65cc9bf52e3ab97683493" gracePeriod=30 Oct 31 23:22:24.595866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:24.595831 5043 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 23:22:24.596243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:24.596193 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 23:22:24.965467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:24.965438 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:25.067958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.067921 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca\") pod \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " Oct 31 23:22:25.068294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.068274 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7\" (UniqueName: \"kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7\") pod \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " Oct 31 23:22:25.068606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.068586 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca\") pod \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\" (UID: \"0e494ca7-3dba-4df4-ad21-fcddadc8121d\") " Oct 31 23:22:25.071210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.071187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca" pod "0e494ca7-3dba-4df4-ad21-fcddadc8121d" (UID: "0e494ca7-3dba-4df4-ad21-fcddadc8121d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:25.079448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.075466 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7" pod "0e494ca7-3dba-4df4-ad21-fcddadc8121d" (UID: "0e494ca7-3dba-4df4-ad21-fcddadc8121d"). InnerVolumeSpecName "kube-api-access-lcrd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:25.079448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.075599 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca" pod "0e494ca7-3dba-4df4-ad21-fcddadc8121d" (UID: "0e494ca7-3dba-4df4-ad21-fcddadc8121d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:22:25.079448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.077625 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g"] Oct 31 23:22:25.079448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.078934 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0e494ca7-3dba-4df4-ad21-fcddadc8121d" containerName="revoke-keys" Oct 31 23:22:25.079448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.078953 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e494ca7-3dba-4df4-ad21-fcddadc8121d" containerName="revoke-keys" Oct 31 23:22:25.079448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.079191 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e494ca7-3dba-4df4-ad21-fcddadc8121d" containerName="revoke-keys" Oct 31 23:22:25.090225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.087765 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.090225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.089927 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g"] Oct 31 23:22:25.092817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.092792 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-836c99-serving-cert\"" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.196845 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198123 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198192 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198340 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9f9c\" (UniqueName: \"kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198489 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198861 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198886 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrd7\" (UniqueName: \"kubernetes.io/projected/0e494ca7-3dba-4df4-ad21-fcddadc8121d-kube-api-access-lcrd7\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:25.202586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.198904 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e494ca7-3dba-4df4-ad21-fcddadc8121d-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:25.300898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.300863 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.301118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.300928 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.301118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.301056 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9f9c\" (UniqueName: \"kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.301257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.301169 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.302158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.302129 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.303886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.303867 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.304269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.304245 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.322838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.318397 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9f9c\" (UniqueName: \"kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c\") pod \"maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.369958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.369919 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:25.511491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.511456 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:25.548788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.548388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:25.564354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.564330 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:22:25.594467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.594436 5043 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 23:22:25.594626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.594500 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 23:22:25.677344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.677308 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" event={"ID":"0e494ca7-3dba-4df4-ad21-fcddadc8121d","Type":"ContainerDied","Data":"29e08121d4da921da4e0c3195bd39490eafc8614125f2b6e42e670e5eb869501"} Oct 31 23:22:25.677533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.677371 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e08121d4da921da4e0c3195bd39490eafc8614125f2b6e42e670e5eb869501" Oct 31 23:22:25.677533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:25.677443 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4" Oct 31 23:22:26.126555 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:26.109484 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b76e285_2b19_4a24_8c4b_fcba4ee82775.slice/crio-1174bdf6334903bf1cca422fefd03c777d6f0abdd2eda10e2d3ddd726aa767eb WatchSource:0}: Error finding container 1174bdf6334903bf1cca422fefd03c777d6f0abdd2eda10e2d3ddd726aa767eb: Status 404 returned error can't find the container with id 1174bdf6334903bf1cca422fefd03c777d6f0abdd2eda10e2d3ddd726aa767eb Oct 31 23:22:26.126555 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.112574 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g"] Oct 31 23:22:26.150792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.132053 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:26.545129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.545089 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh"] Oct 31 23:22:26.550456 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.550432 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.566857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.566835 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-270e8c-openshift-default-dockercfg-sbdqn\"" Oct 31 23:22:26.580190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.580153 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4"] Oct 31 23:22:26.585247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.585224 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.594810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.594778 5043 patch_prober.go:29] interesting pod/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 31 23:22:26.594915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.594820 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 31 23:22:26.605690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.605663 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh"] Oct 31 23:22:26.653842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.653813 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4"] Oct 31 23:22:26.683841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.683809 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.684058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.683869 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7h5\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.684211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684118 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684162 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99f2\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684249 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.684328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684284 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684364 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684402 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.684524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684465 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684489 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684576 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684617 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.684916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684712 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.685054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684933 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.685054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.684979 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.685054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.685003 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.685233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.685062 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.685233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.685089 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.692373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.692341 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" event={"ID":"7b76e285-2b19-4a24-8c4b-fcba4ee82775","Type":"ContainerStarted","Data":"1174bdf6334903bf1cca422fefd03c777d6f0abdd2eda10e2d3ddd726aa767eb"} Oct 31 23:22:26.786989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.786962 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787005 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787059 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787085 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7h5\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787148 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787169 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99f2\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787198 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787221 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787259 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787301 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787524 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787617 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787644 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787706 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.787843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787790 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.787843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787834 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.788205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787883 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.788205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.787903 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.788321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.788261 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.788432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.788411 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.788518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.788468 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.788826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.788802 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.788997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.788973 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.789109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.789088 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.789191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.789167 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.789453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.789430 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.789533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.789496 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.790147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.790128 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.790752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.790624 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.791177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.791140 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.791314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.791288 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.792830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.792811 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.813140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.813089 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7h5\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.813270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.813248 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.861717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.861653 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.864294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.863853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99f2\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.875036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.875002 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:26.884175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.884148 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm"] Oct 31 23:22:26.908339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.902863 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:26.908339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.905802 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.956654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.956130 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-885fdf-openshift-default-dockercfg-9mwfb\"" Oct 31 23:22:26.957443 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.957421 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl"] Oct 31 23:22:26.965024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.965002 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:26.967132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.967108 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm"] Oct 31 23:22:26.976649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.976612 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl"] Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.993781 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.993974 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.994004 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.994029 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.994080 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.994102 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.995050 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxj8\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.995178 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:26.995630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:26.995223 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098113 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098196 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098235 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098262 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098303 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098337 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098378 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098400 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098423 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjgkw\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098439 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098568 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098630 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxj8\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098700 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098745 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098813 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098843 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.098940 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.099002 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.101860 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.102317 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.102359 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.102788 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.109469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.102853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.137531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.112054 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.137531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.137378 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.168502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.168401 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxj8\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.168502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.168448 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.200640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.200597 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.200839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.200688 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.200839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.200756 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgkw\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.200839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.200785 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.201007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.200887 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.201253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.201059 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.201253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.201198 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.201285 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.201325 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.201739 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.202191 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.202520 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.202808 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.203199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.202842 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.204789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.204722 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.206995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.206540 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.249447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.245144 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.249447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.249340 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgkw\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.308983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.308930 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:27.381753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.381647 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh"] Oct 31 23:22:27.402166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.402125 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4"] Oct 31 23:22:27.402520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.402493 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:27.410709 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:27.410684 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6bd84_4b59_4823_8613_8b6519a6db0d.slice/crio-12d28136467e1b184b5a59ff55c949480777fcd0ece2249e6888a69b1ad82349 WatchSource:0}: Error finding container 12d28136467e1b184b5a59ff55c949480777fcd0ece2249e6888a69b1ad82349: Status 404 returned error can't find the container with id 12d28136467e1b184b5a59ff55c949480777fcd0ece2249e6888a69b1ad82349 Oct 31 23:22:27.411196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.411159 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.411265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.411249 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.411315 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.411295 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.421389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.420602 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.421389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.420698 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.421389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.420742 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.459777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.459707 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:27.738469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.702526 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:27.738469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.713034 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:27.748102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.748031 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:22:27.769751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.769679 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm"] Oct 31 23:22:27.772751 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:27.772670 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9981a666_c637_4f4d_8f48_6e58b9172543.slice/crio-7a75973d460039856ea3299e8eccac4a20c9061285f356f45ac5032550200021 WatchSource:0}: Error finding container 7a75973d460039856ea3299e8eccac4a20c9061285f356f45ac5032550200021: Status 404 returned error can't find the container with id 7a75973d460039856ea3299e8eccac4a20c9061285f356f45ac5032550200021 Oct 31 23:22:27.780200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.778881 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" event={"ID":"7b76e285-2b19-4a24-8c4b-fcba4ee82775","Type":"ContainerStarted","Data":"80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a"} Oct 31 23:22:27.780200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.779274 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:27.782100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.780842 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.782100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.780921 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.782100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.780959 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.797954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.797901 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" event={"ID":"09b6bd84-4b59-4823-8613-8b6519a6db0d","Type":"ContainerStarted","Data":"4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5"} Oct 31 23:22:27.798189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.798164 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" event={"ID":"09b6bd84-4b59-4823-8613-8b6519a6db0d","Type":"ContainerStarted","Data":"12d28136467e1b184b5a59ff55c949480777fcd0ece2249e6888a69b1ad82349"} Oct 31 23:22:27.803286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.802067 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" event={"ID":"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4","Type":"ContainerStarted","Data":"bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9"} Oct 31 23:22:27.803286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.802171 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" event={"ID":"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4","Type":"ContainerStarted","Data":"3eea2833f04aac4c1fe6d079503e87706000894ef93ef8ca27d3ddc04811768c"} Oct 31 23:22:27.875624 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:27.875592 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe644d97_00b2_4a09_8917_173f8c866aca.slice/crio-afb7b1ed281c273b3b0b873470d658294c36ac9301219ec07d6f46760280ade7 WatchSource:0}: Error finding container afb7b1ed281c273b3b0b873470d658294c36ac9301219ec07d6f46760280ade7: Status 404 returned error can't find the container with id afb7b1ed281c273b3b0b873470d658294c36ac9301219ec07d6f46760280ade7 Oct 31 23:22:27.878419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.878342 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:27.879962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.879140 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" podStartSLOduration=2.300016361 podStartE2EDuration="2.879122471s" totalImagesPullingTime="579.10611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:27.845077081 +0000 UTC m=+1974.074631500" watchObservedRunningTime="2026-10-31 23:22:27.879122471 +0000 UTC m=+1974.108676885" Oct 31 23:22:27.879962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.879425 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.879962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.879488 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.879962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.879584 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:27.890732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.890530 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl"] Oct 31 23:22:27.890732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.890650 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podStartSLOduration=1.8906268179999999 podStartE2EDuration="1.890626818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:27.874738597 +0000 UTC m=+1974.104293037" watchObservedRunningTime="2026-10-31 23:22:27.890626818 +0000 UTC m=+1974.120181230" Oct 31 23:22:27.923924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:27.909266 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:28.197090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.197008 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:28.473687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.473591 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:28.548253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.548185 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podStartSLOduration=2.548163713 podStartE2EDuration="2.548163713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:27.917444679 +0000 UTC m=+1974.146999092" watchObservedRunningTime="2026-10-31 23:22:28.548163713 +0000 UTC m=+1974.777718124" Oct 31 23:22:28.671574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.660060 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:28.688641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.688266 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct"] Oct 31 23:22:28.762725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.757907 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36a15fc5-twvct"] Oct 31 23:22:28.818230 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.812511 5043 scope.go:122] "RemoveContainer" containerID="0b2d8cb318887aac6b799ba1c9322eed3426b35b4c726d09cf3a18dd26d6f7e8" Oct 31 23:22:28.961676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.958162 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:28.961676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.958237 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:28.962184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.962075 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:28.962184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.962125 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:28.986051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.986013 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" event={"ID":"be644d97-00b2-4a09-8917-173f8c866aca","Type":"ContainerStarted","Data":"725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15"} Oct 31 23:22:28.986336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:28.986310 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" event={"ID":"be644d97-00b2-4a09-8917-173f8c866aca","Type":"ContainerStarted","Data":"afb7b1ed281c273b3b0b873470d658294c36ac9301219ec07d6f46760280ade7"} Oct 31 23:22:29.000950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.000873 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" event={"ID":"9981a666-c637-4f4d-8f48-6e58b9172543","Type":"ContainerStarted","Data":"ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645"} Oct 31 23:22:29.001201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.001179 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" event={"ID":"9981a666-c637-4f4d-8f48-6e58b9172543","Type":"ContainerStarted","Data":"7a75973d460039856ea3299e8eccac4a20c9061285f356f45ac5032550200021"} Oct 31 23:22:29.007249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.007227 5043 scope.go:122] "RemoveContainer" containerID="e29ef966fe50e855687948706d2a78d30d1f4e59383978322d4d6003ad55b408" Oct 31 23:22:29.055633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.051508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:29.231184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.223385 5043 scope.go:122] "RemoveContainer" containerID="35469d41d88c962f7545fe8df512a334b4e1c24bd26d8c77412b355fac00cf50" Oct 31 23:22:29.289889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.289852 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:29.293439 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.293381 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podStartSLOduration=3.293358983 podStartE2EDuration="3.293358983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:29.019924401 +0000 UTC m=+1975.249478832" watchObservedRunningTime="2026-10-31 23:22:29.293358983 +0000 UTC m=+1975.522913397" Oct 31 23:22:29.686685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.686635 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:29.771726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.715682 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:29.975775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:29.975674 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:30.257570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.254864 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:30.291571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.285658 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:30.301568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.300529 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.301568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.300613 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:30.301568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.301241 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:30.320565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.319024 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.320565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.319084 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:30.381572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.358274 5043 scope.go:122] "RemoveContainer" containerID="ea5e2074eae85308b0fb671db43f7d3af0f142f1cd30c751b8dcb25c66300f6a" Oct 31 23:22:30.381572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.379495 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-55lxc" podUID="9cf1ba00-1cc7-4af1-a5df-efeafadaac9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded" Oct 31 23:22:30.387761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.383767 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:30.408722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.408029 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.408722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.408081 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:30.429427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.429384 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.429714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.429686 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:30.478483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.478212 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.478800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.478770 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:30.556711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.525689 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podStartSLOduration=4.52566749 podStartE2EDuration="4.52566749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:29.222437876 +0000 UTC m=+1975.451992288" watchObservedRunningTime="2026-10-31 23:22:30.52566749 +0000 UTC m=+1976.755221903" Oct 31 23:22:30.679567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.676586 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:30.736585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.704162 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28549d8a-85b5-4060-b623-be279fa8d14f" path="/var/lib/kubelet/pods/28549d8a-85b5-4060-b623-be279fa8d14f/volumes" Oct 31 23:22:30.881710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.881671 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.882327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.882293 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:30.898169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.897742 5043 scope.go:122] "RemoveContainer" containerID="3d8fadd7f7cde659665f33db449490030d5933036832b9df1eab3dc50d79cd5e" Oct 31 23:22:30.906879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.906400 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:30.906879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:30.906458 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:31.117628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.106958 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:31.170387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.169911 5043 scope.go:122] "RemoveContainer" containerID="2e8a1d64c32b3211512abbc141fd85532390fb08da123af970a805af5a6f0692" Oct 31 23:22:31.276651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.276008 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:31.278353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.276896 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l"] Oct 31 23:22:31.278353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.277000 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:22:31.297986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.295098 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:31.297986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.296060 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:31.297986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.296477 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:22:31.313610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.304655 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:31.313610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.305176 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:22:31.434763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.434641 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:31.480777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.477407 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:31.480777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.477468 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:31.480777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.480673 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:31.480777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.480720 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:31.529818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.529286 5043 scope.go:122] "RemoveContainer" containerID="d92b844ec596fa77e27176e042eca461aad29ad582463d39277820c3fc5f9018" Oct 31 23:22:31.688626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.687705 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:22:31.846566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:31.846262 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.105765 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.105828 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.106243 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.106290 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.106808 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.191229 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.244258 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:32.284789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.282310 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:32.356570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.346257 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:32.423109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.423035 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:32.423906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.423827 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:32.446571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.445409 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:32.495958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.495922 5043 scope.go:122] "RemoveContainer" containerID="04c905332395c12b004e61419e22a2cd7d145302a9ac34e17ff50b5eab4c4223" Oct 31 23:22:32.526870 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.508614 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:32.547628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.527086 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:32.763823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.763770 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.792624 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.792697 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.792767 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.792840 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.792949 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.792999 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.793035 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.793213 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.793380 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.797478 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.797522 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b407cae8-6zt6l"] Oct 31 23:22:32.802623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.797600 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:22:32.949722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.949676 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:33.048818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:32.998656 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:33.048818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.007616 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:33.048818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.012704 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:33.048818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.007015 5043 scope.go:122] "RemoveContainer" containerID="d54ffc6e9ecda7b1d91616d4312030722ab605b99e491cb4995a096e9ffd1ea6" Oct 31 23:22:33.107636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.091508 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:33.107636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.091623 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:33.122607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.121389 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:33.149442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.131726 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:33.149442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.131802 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:33.173882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.151880 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 31 23:22:33.173882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.163016 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:33.173882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.163055 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:33.173882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.163072 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:33.173882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.163088 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:33.173882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.163110 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/433ee218-d3e7-4837-98f8-642a6266fdf9-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:33.229607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.207874 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "kube-api-access-d6mv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:33.311585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.304059 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:22:33.348988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.348935 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:33.367925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.357656 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:33.486441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.486396 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:33.487260 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.487139 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:33.491772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.491750 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6mv8\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-kube-api-access-d6mv8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:33.545141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.544899 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:22:33.591426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.591381 5043 generic.go:356] "Generic (PLEG): container finished" podID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerID="96003ef535034fb9211ba89a8bbf1708a2d9944b47b65cc9bf52e3ab97683493" exitCode=0 Oct 31 23:22:34.061715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:33.597950 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" event={"ID":"433ee218-d3e7-4837-98f8-642a6266fdf9","Type":"ContainerDied","Data":"96003ef535034fb9211ba89a8bbf1708a2d9944b47b65cc9bf52e3ab97683493"} Oct 31 23:22:34.235607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.187302 5043 scope.go:122] "RemoveContainer" containerID="96003ef535034fb9211ba89a8bbf1708a2d9944b47b65cc9bf52e3ab97683493" Oct 31 23:22:34.235607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.200779 5043 scope.go:122] "RemoveContainer" containerID="c80e9eb07b05228715ce9d5fb8f1ea4fc46c76023c0c99f5b2867ca1725362a9" Oct 31 23:22:34.304708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.304653 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:34.378629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.351426 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:22:34.433571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.353803 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" Oct 31 23:22:34.453079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.453041 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:34.472105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.471475 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:22:34.474143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.474108 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:34.491384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.491343 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:34.491603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.491419 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:34.620185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.604844 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:34.660615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.658209 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\") pod \"433ee218-d3e7-4837-98f8-642a6266fdf9\" (UID: \"433ee218-d3e7-4837-98f8-642a6266fdf9\") " Oct 31 23:22:34.660615 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:34.659850 5043 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/433ee218-d3e7-4837-98f8-642a6266fdf9/volumes/kubernetes.io~projected/istio-token Oct 31 23:22:34.660615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.659946 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:34.682785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.682748 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:34.827231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.827186 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:34.831435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.831398 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:34.832442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.832402 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.589695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.589654 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:35.604880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.865796 5043 scope.go:122] "RemoveContainer" containerID="5d27d729108bc3f61da8492942764f57bdedfe471909d91a2e519b823497c6c0" Oct 31 23:22:35.605235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.374273 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:22:35.606181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.606130 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:22:35.606795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.606761 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:22:35.620601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.614860 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:22:35.624633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.624587 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:35.643689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.643654 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/433ee218-d3e7-4837-98f8-642a6266fdf9-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:35.645799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.645769 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.653610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.646039 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:35.738767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.992827 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert" pod "433ee218-d3e7-4837-98f8-642a6266fdf9" (UID: "433ee218-d3e7-4837-98f8-642a6266fdf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:35.739010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.053473 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.739197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.739154 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:35.767667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.711962 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-spffz" podUID="d63e791b-7230-4884-bdba-3996f4205f60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:35.825095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.793343 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:22:35.845742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:34.866046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:35.846115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.693387 5043 scope.go:122] "RemoveContainer" containerID="948f0c81bdb8dde619333b40e0c0b1ec042451eb4deab727cb70e6c10f12d08b" Oct 31 23:22:35.846450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.099583 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.846751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.846706 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:35.846926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.188171 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.847075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.847054 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:35.873123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.873078 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.948569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.940000 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:35.948569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.945248 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:35.948569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.945305 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:36.011397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.984593 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/433ee218-d3e7-4837-98f8-642a6266fdf9-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:36.303684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.993033 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:36.360572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.359833 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:36.360817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:35.993432 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:36.442318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.425494 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:36.485121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.485082 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:36.485430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.485381 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:36.525264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.525228 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:36.525608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.525578 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:36.632019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.631739 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:36.649748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.649706 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:22:36.651188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.651158 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 23:22:36.826979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.826934 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3113e8e9-2590-406b-a181-5b52bad97a12" path="/var/lib/kubelet/pods/3113e8e9-2590-406b-a181-5b52bad97a12/volumes" Oct 31 23:22:36.867765 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:36.863530 5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.801s" Oct 31 23:22:36.901630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.565345 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:36.926112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.902792 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:36.926112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.902858 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:36.971579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.950842 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:37.011577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.011512 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:37.011854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.992328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:37.012105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.902221 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:22:37.012682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:36.995592 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:22:37.292931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.231071 5043 scope.go:122] "RemoveContainer" containerID="40e43c90b5d67053b96b650f6cd994979e2aef0a539c7a2bfadf35669e21f1e3" Oct 31 23:22:37.292931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.234418 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:37.445903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.421273 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:22:37.445903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.421359 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:22:37.445903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.421447 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6" event={"ID":"433ee218-d3e7-4837-98f8-642a6266fdf9","Type":"ContainerDied","Data":"bfca8a6e0741e2582db5d7cca8d179673656826dfcfc2a33a6c70628d4bb47c9"} Oct 31 23:22:37.682863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:37.682760 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:22:38.049569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.034758 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:38.049569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.034855 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:38.199694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.199652 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:22:38.221073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.221028 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 31 23:22:38.355639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.343908 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 23:22:38.359569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.355924 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 23:22:38.455576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.413131 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 31 23:22:38.511243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.511197 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 31 23:22:38.534214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.534163 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 23:22:38.534534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.534493 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 23:22:38.534701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.505277 5043 scope.go:122] "RemoveContainer" containerID="fd08310dfb31c18534e922f8f739bac67016207d1fe8942b252a3e7fd51cbd70" Oct 31 23:22:38.570901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.504492 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 23:22:38.571205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.571164 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 23:22:38.571688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.460644 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:22:38.574883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.460711 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:38.581530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.581420 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:38.598804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.598770 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6"] Oct 31 23:22:38.599110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.599097 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:22:38.599234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.599225 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-r2kv6"] Oct 31 23:22:38.630655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.625825 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:38.630655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.627723 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:38.632920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.631702 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:22:38.644572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.644528 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:22:38.649419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.649396 5043 scope.go:122] "RemoveContainer" containerID="fd4e4199fe4e90633e8994d445ddb7fabfc0e18bdd26845ce6a61e48569db1e7" Oct 31 23:22:38.683378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.683123 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:22:38.693138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.693109 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:22:38.697077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.697035 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:22:38.731123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.731057 5043 scope.go:122] "RemoveContainer" containerID="95ad86dc72a7c50f2df2720af645ae7eaa169d811aad386bf2a1a41632c26e53" Oct 31 23:22:38.757770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.757746 5043 scope.go:122] "RemoveContainer" containerID="b338d82c0b906c30d6ac55ae97fc8626e6df193597e361a5bfe39359cf9c4f11" Oct 31 23:22:38.886200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.886121 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:38.886843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.886690 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:38.888216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.888156 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:22:38.910065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.910041 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:38.910378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.910362 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:38.911221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:38.911191 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:22:39.331131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:39.331091 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:39.428903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:39.424631 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:39.428903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:39.426359 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:39.431188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:39.431126 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:22:39.943214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:39.943115 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" path="/var/lib/kubelet/pods/433ee218-d3e7-4837-98f8-642a6266fdf9/volumes" Oct 31 23:22:40.009654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:40.009622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:40.211681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:40.205698 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:40.461630 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:40.461591 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:22:40.946573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:40.946161 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:41.651936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:41.651890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:41.923629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:41.922877 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:42.044571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.044215 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s"] Oct 31 23:22:42.064565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.061293 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a16dfb-qvt8s"] Oct 31 23:22:42.095568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.095082 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:42.323893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.323854 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:42.776071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.776025 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:42.869257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.869220 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb"] Oct 31 23:22:42.870183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.870159 5043 cpu_manager.go:414] "containerMap: removing container" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" Oct 31 23:22:42.870183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.870182 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" Oct 31 23:22:42.870464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.870448 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" Oct 31 23:22:42.880012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.879989 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.887056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.887032 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb"] Oct 31 23:22:42.915643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.914678 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-479107-openshift-default-dockercfg-pnn5x\"" Oct 31 23:22:42.968136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968098 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.968292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968222 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.968413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968379 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.968622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968491 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.968702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968664 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.968702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968698 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.968912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.968887 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqk9c\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.969102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.969088 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:42.969242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:42.969210 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071309 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071351 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071486 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk9c\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071580 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071620 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071744 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.071898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071797 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.072127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.071919 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.072127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.072080 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.072218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.072149 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.072271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.072237 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.072657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.072632 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.072826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.072805 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.073862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.073834 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.074136 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.074114 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.100937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.100913 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk9c\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.103069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.103049 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token\") pod \"e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.198680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.198631 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:43.429669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.404148 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:43.654470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.650703 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:43.799120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.799085 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:43.943481 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:43.942807 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849be6ec_37ac_48a0_b865_844a8fcf8f10.slice/crio-72f6bdbc3be59d179094c3ea76d569bc3ea0e253e26d0df800c77178c3c64967 WatchSource:0}: Error finding container 72f6bdbc3be59d179094c3ea76d569bc3ea0e253e26d0df800c77178c3c64967: Status 404 returned error can't find the container with id 72f6bdbc3be59d179094c3ea76d569bc3ea0e253e26d0df800c77178c3c64967 Oct 31 23:22:43.957418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.957383 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47687c36-f155-4842-b818-5f5133494786" path="/var/lib/kubelet/pods/47687c36-f155-4842-b818-5f5133494786/volumes" Oct 31 23:22:43.957966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.957944 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb"] Oct 31 23:22:43.958896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.958814 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:43.959009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.958953 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:43.959009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:43.959002 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:22:44.018409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:44.018380 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" event={"ID":"849be6ec-37ac-48a0-b865-844a8fcf8f10","Type":"ContainerStarted","Data":"72f6bdbc3be59d179094c3ea76d569bc3ea0e253e26d0df800c77178c3c64967"} Oct 31 23:22:44.750571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:44.750496 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:45.060611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:45.060489 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" event={"ID":"849be6ec-37ac-48a0-b865-844a8fcf8f10","Type":"ContainerStarted","Data":"c6dffb4bc897d66cc9d638d13d4a567588ca65958fda26121828e98a37966319"} Oct 31 23:22:45.089423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:45.089373 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podStartSLOduration=3.089359628 podStartE2EDuration="3.089359628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:45.087668247 +0000 UTC m=+1991.317222661" watchObservedRunningTime="2026-10-31 23:22:45.089359628 +0000 UTC m=+1991.318914039" Oct 31 23:22:45.199420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:45.199388 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:45.201054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:45.201028 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 23:22:45.201185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:45.201071 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 23:22:45.603415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:45.603296 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:46.199917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:46.199880 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 23:22:46.200310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:46.199940 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 23:22:46.471746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:46.471644 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:46.765908 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:46.765868 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:22:47.212809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:47.212711 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 23:22:47.213487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:47.213455 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 23:22:47.857580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:47.857114 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:48.051587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.050523 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:48.206157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.206065 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 23:22:48.206157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.206136 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 23:22:48.252572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.249195 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:48.399510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.399424 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp"] Oct 31 23:22:48.400086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.400031 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" containerID="cri-o://dc71b80747ace761be5e652bcbcb219230b36bc09e066e6b83a1ab2c150d4b93" gracePeriod=30 Oct 31 23:22:48.412333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.412298 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf"] Oct 31 23:22:48.432442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.432236 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx"] Oct 31 23:22:48.484829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.484702 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" containerID="cri-o://75bda808064c36c6b90b000fde31553ae9897010a81d8e284434457a9f1d6237" gracePeriod=30 Oct 31 23:22:48.486310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.485268 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" containerID="cri-o://31b604bd4cc786af918d2adbe3afc592bc2636bad0bd2e63c399d07d4578c69d" gracePeriod=30 Oct 31 23:22:48.498393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:48.498325 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:49.177333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.177291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:49.199933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.199900 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 23:22:49.200108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.199952 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 23:22:49.275008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.274432 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp"] Oct 31 23:22:49.276162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.275826 5043 cpu_manager.go:414] "containerMap: removing container" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" Oct 31 23:22:49.276162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.275855 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" Oct 31 23:22:49.276162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.276160 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="433ee218-d3e7-4837-98f8-642a6266fdf9" containerName="istio-proxy" Oct 31 23:22:49.284364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.282080 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.293732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.293619 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bf279c-dockercfg-54tb6\"" Oct 31 23:22:49.293898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.293778 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp"] Oct 31 23:22:49.296983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.296278 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bf279c\"" Oct 31 23:22:49.372610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.369990 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:49.373975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.373939 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5bk\" (UniqueName: \"kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk\") pod \"payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.374128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.374111 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.477417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.476998 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5bk\" (UniqueName: \"kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk\") pod \"payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.477417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.477173 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.518572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.481697 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.535765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.535691 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:49.536831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.536805 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5bk\" (UniqueName: \"kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk\") pod \"payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:49.613202 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:49.613164 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:50.192032 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:22:50.191997 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb57d425_439d_4b90_aa22_11d8960fc37a.slice/crio-102a53b5e701ec28a4c10034f5d830f7d8d8e54851fa8a22fb22fec936c48813 WatchSource:0}: Error finding container 102a53b5e701ec28a4c10034f5d830f7d8d8e54851fa8a22fb22fec936c48813: Status 404 returned error can't find the container with id 102a53b5e701ec28a4c10034f5d830f7d8d8e54851fa8a22fb22fec936c48813 Oct 31 23:22:50.193595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:50.193565 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp"] Oct 31 23:22:50.200020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:50.199994 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 23:22:50.200129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:50.200041 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 23:22:50.285616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:50.282767 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:50.519571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:50.517055 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:50.967813 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:50.967681 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:22:51.199337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.199294 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" event={"ID":"fb57d425-439d-4b90-aa22-11d8960fc37a","Type":"ContainerStarted","Data":"c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d"} Oct 31 23:22:51.199584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.199363 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:22:51.199584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.199388 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" event={"ID":"fb57d425-439d-4b90-aa22-11d8960fc37a","Type":"ContainerStarted","Data":"102a53b5e701ec28a4c10034f5d830f7d8d8e54851fa8a22fb22fec936c48813"} Oct 31 23:22:51.223622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.217667 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:51.223622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.219094 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:51.249567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.248840 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:22:51.275585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.265860 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:51.280602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:51.275772 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" podStartSLOduration=2.275750455 podStartE2EDuration="2.275750455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:22:51.231052468 +0000 UTC m=+1997.460606880" watchObservedRunningTime="2026-10-31 23:22:51.275750455 +0000 UTC m=+1997.505304867" Oct 31 23:22:51.965567 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:51.965021 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:22:51.965567 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:51.965066 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:22:52.664568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:52.647572 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:53.648049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:53.639294 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:54.705861 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.705815 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:54.743572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.735645 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 31 23:22:54.759613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.747104 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 31 23:22:54.826591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.804739 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:54.862636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.830164 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:54.878378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.878341 5043 generic.go:356] "Generic (PLEG): container finished" podID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerID="31b604bd4cc786af918d2adbe3afc592bc2636bad0bd2e63c399d07d4578c69d" exitCode=0 Oct 31 23:22:54.884283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.884252 5043 generic.go:356] "Generic (PLEG): container finished" podID="b2d8b699-311d-48f7-be78-2223ba357359" containerID="dc71b80747ace761be5e652bcbcb219230b36bc09e066e6b83a1ab2c150d4b93" exitCode=0 Oct 31 23:22:54.889044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.889020 5043 generic.go:356] "Generic (PLEG): container finished" podID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerID="75bda808064c36c6b90b000fde31553ae9897010a81d8e284434457a9f1d6237" exitCode=0 Oct 31 23:22:54.962516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.962432 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" event={"ID":"e3f3e249-1514-4c28-a06f-d7db747a4ef9","Type":"ContainerDied","Data":"31b604bd4cc786af918d2adbe3afc592bc2636bad0bd2e63c399d07d4578c69d"} Oct 31 23:22:54.962789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.962765 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" event={"ID":"b2d8b699-311d-48f7-be78-2223ba357359","Type":"ContainerDied","Data":"dc71b80747ace761be5e652bcbcb219230b36bc09e066e6b83a1ab2c150d4b93"} Oct 31 23:22:54.962938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:54.962923 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" event={"ID":"3a9dcb05-0ae4-4b58-bed7-39549e5800cd","Type":"ContainerDied","Data":"75bda808064c36c6b90b000fde31553ae9897010a81d8e284434457a9f1d6237"} Oct 31 23:22:55.078668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.078618 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:55.229205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.229088 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:55.675571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.667016 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:22:55.794471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.786440 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:22:55.819657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.814361 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.834624 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.834686 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.834908 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.835030 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.835062 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.835109 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.835220 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.835263 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.859681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.835354 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo\") pod \"b2d8b699-311d-48f7-be78-2223ba357359\" (UID: \"b2d8b699-311d-48f7-be78-2223ba357359\") " Oct 31 23:22:55.872493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.871965 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:55.873315 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.873287 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:55.925206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.875740 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:55.925206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.883496 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:55.925206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.894916 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:55.925206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.899713 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "kube-api-access-5b92f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:55.983622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.981923 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:55.991025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.990733 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d8b699-311d-48f7-be78-2223ba357359-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:55.991258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.991240 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:55.991854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.991830 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b92f\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-kube-api-access-5b92f\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:55.992013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.991999 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:55.992107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.992095 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:55.992201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.992190 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:55.992284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:55.992273 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d8b699-311d-48f7-be78-2223ba357359-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.039409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.022410 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:22:56.150919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.150807 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.152132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.152102 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.153965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.153942 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.155092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.154669 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.155697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.155667 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.159119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.159099 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.159292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.159276 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.160496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.159862 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.160496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.159981 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token" pod "b2d8b699-311d-48f7-be78-2223ba357359" (UID: "b2d8b699-311d-48f7-be78-2223ba357359"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:56.160860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.160821 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.161001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.160987 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data\") pod \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\" (UID: \"3a9dcb05-0ae4-4b58-bed7-39549e5800cd\") " Oct 31 23:22:56.162350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.162329 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.168132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.167938 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.168512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.168264 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.170021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.168674 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.170021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.169172 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" event={"ID":"3a9dcb05-0ae4-4b58-bed7-39549e5800cd","Type":"ContainerDied","Data":"0fbe55994721041c7a6ebdfa67a2f6cccc00d89d08eefb4fb1153185c740b2dc"} Oct 31 23:22:56.170021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.169234 5043 scope.go:122] "RemoveContainer" containerID="75bda808064c36c6b90b000fde31553ae9897010a81d8e284434457a9f1d6237" Oct 31 23:22:56.175569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.175168 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:56.175569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.175512 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" event={"ID":"e3f3e249-1514-4c28-a06f-d7db747a4ef9","Type":"ContainerDied","Data":"611b0be09330c24d5463b4ff973a576aceb3ffb71502e993552a95818920699a"} Oct 31 23:22:56.175849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.175723 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx" Oct 31 23:22:56.179526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.156275 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf" Oct 31 23:22:56.180014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.179995 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.182872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.182848 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" event={"ID":"b2d8b699-311d-48f7-be78-2223ba357359","Type":"ContainerDied","Data":"8d36526d60e14a955eb138e33b129060d2f8f0bd160c70196d5136e7d112e71e"} Oct 31 23:22:56.183107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.183089 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp" Oct 31 23:22:56.187062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.187040 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:56.202941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.202900 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.300665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.297928 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:56.367661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.362131 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:56.370872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.370840 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.378700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.378671 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.378933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.378913 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.409078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.409001 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.380227 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.423484 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.423500 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.423563 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.433649 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.433709 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.457570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.433886 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token\") pod \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\" (UID: \"e3f3e249-1514-4c28-a06f-d7db747a4ef9\") " Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462791 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462826 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462841 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462855 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462870 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d8b699-311d-48f7-be78-2223ba357359-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462884 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462898 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462911 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462925 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.462942 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d8b699-311d-48f7-be78-2223ba357359-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.493701 5043 scope.go:122] "RemoveContainer" containerID="31b604bd4cc786af918d2adbe3afc592bc2636bad0bd2e63c399d07d4578c69d" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.494247 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.494575 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:22:56.507944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.496645 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.549470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.510661 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:56.549470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.534137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:56.549470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.539738 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp" pod "3a9dcb05-0ae4-4b58-bed7-39549e5800cd" (UID: "3a9dcb05-0ae4-4b58-bed7-39549e5800cd"). InnerVolumeSpecName "kube-api-access-t8wqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:56.576582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.550253 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:56.576582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.576124 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.576582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.576159 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wqp\" (UniqueName: \"kubernetes.io/projected/3a9dcb05-0ae4-4b58-bed7-39549e5800cd-kube-api-access-t8wqp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.576582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.576176 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.576582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.576192 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.576582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.576206 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:56.705310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.669362 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:56.705310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:56.701518 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:57.065575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.059442 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:22:57.066491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.066456 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:57.133570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.090883 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "kube-api-access-rqxm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:22:57.133570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.105227 5043 scope.go:122] "RemoveContainer" containerID="dc71b80747ace761be5e652bcbcb219230b36bc09e066e6b83a1ab2c150d4b93" Oct 31 23:22:57.192713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.182427 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo" pod "e3f3e249-1514-4c28-a06f-d7db747a4ef9" (UID: "e3f3e249-1514-4c28-a06f-d7db747a4ef9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:22:57.571715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.569500 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:57.571715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.569904 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:22:57.846392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.841723 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:57.876703 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.876651 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:57.907193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.907159 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:57.907470 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.907446 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqxm6\" (UniqueName: \"kubernetes.io/projected/e3f3e249-1514-4c28-a06f-d7db747a4ef9-kube-api-access-rqxm6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:57.944588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.927647 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f3e249-1514-4c28-a06f-d7db747a4ef9-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:22:57.944588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.928141 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp"] Oct 31 23:22:57.944588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:57.928755 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:58.257429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.257377 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:58.482584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.479595 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-479107-zlhhp"] Oct 31 23:22:58.482584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.479650 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp"] Oct 31 23:22:58.482584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.479677 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-xqqgp"] Oct 31 23:22:58.486523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.486489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:58.565590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.565473 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:58.579685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.579642 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 31 23:22:58.631672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.631624 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:58.684712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.684608 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf"] Oct 31 23:22:58.751768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.751726 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-chwmf"] Oct 31 23:22:58.801632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.801588 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:58.891784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.891603 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx"] Oct 31 23:22:58.893999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.893970 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:58.949440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.932863 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:22:58.973574 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:22:58.936275 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9dcb05_0ae4_4b58_bed7_39549e5800cd.slice/crio-0fbe55994721041c7a6ebdfa67a2f6cccc00d89d08eefb4fb1153185c740b2dc\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3e249_1514_4c28_a06f_d7db747a4ef9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3e249_1514_4c28_a06f_d7db747a4ef9.slice/crio-611b0be09330c24d5463b4ff973a576aceb3ffb71502e993552a95818920699a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9dcb05_0ae4_4b58_bed7_39549e5800cd.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:22:58.987226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.972932 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:22:58.987226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:58.985983 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a16dfb-openshift-default-57f9ccb585-26ztx"] Oct 31 23:22:59.298308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.298224 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:22:59.298972 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.298602 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:22:59.850570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.849533 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:22:59.980063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.980031 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" path="/var/lib/kubelet/pods/3a9dcb05-0ae4-4b58-bed7-39549e5800cd/volumes" Oct 31 23:22:59.980846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.980827 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d19494-8b02-4c36-a383-8383d45a933c" path="/var/lib/kubelet/pods/65d19494-8b02-4c36-a383-8383d45a933c/volumes" Oct 31 23:22:59.981476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.981458 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d8b699-311d-48f7-be78-2223ba357359" path="/var/lib/kubelet/pods/b2d8b699-311d-48f7-be78-2223ba357359/volumes" Oct 31 23:22:59.982402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:22:59.982386 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" path="/var/lib/kubelet/pods/e3f3e249-1514-4c28-a06f-d7db747a4ef9/volumes" Oct 31 23:23:00.570153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:00.570082 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:01.312647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:01.312601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:01.479684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:01.478920 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj"] Oct 31 23:23:01.498738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:01.498698 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr"] Oct 31 23:23:01.571233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:01.571103 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:01.721433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:01.721329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:02.210018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:02.209874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:02.314390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:02.313428 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:23:02.897421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:02.897263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:03.037711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:03.037667 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:23:03.037908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:03.037740 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:03.557075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:03.556832 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:03.719740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:03.719700 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:23:03.719934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:03.719775 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:04.380858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:04.380817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:04.702842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:04.702748 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:04.848623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:04.848501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:05.054667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:05.054624 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb"] Oct 31 23:23:05.056068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:05.055711 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" containerID="cri-o://c6dffb4bc897d66cc9d638d13d4a567588ca65958fda26121828e98a37966319" gracePeriod=30 Oct 31 23:23:05.127976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:05.127935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:05.299585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:05.299409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:05.495507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:05.493201 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:05.848930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:05.846814 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:06.192508 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:06.192430 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:23:06.222369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.222240 5043 patch_prober.go:29] interesting pod/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:23:06.222586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.222379 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:23:06.479310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.479191 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" podUID="51bcca4d-52c3-4681-a8c9-f7e090ae567a" containerName="payload-processing" containerID="cri-o://25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43" gracePeriod=25 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.500601 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" podUID="f4a81084-193c-40f0-abda-964c909230c0" containerName="payload-pre-processing" containerID="cri-o://c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6" gracePeriod=25 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.622906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.895456 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896522 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896540 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896584 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896590 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896619 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896625 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896744 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9dcb05-0ae4-4b58-bed7-39549e5800cd" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896759 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d8b699-311d-48f7-be78-2223ba357359" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.896769 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f3e249-1514-4c28-a06f-d7db747a4ef9" containerName="istio-proxy" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.958724 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.958757 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.968964 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:06.987669 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.055947 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs\" (UniqueName: \"kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs\") pod \"f4a81084-193c-40f0-abda-964c909230c0\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056007 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume\") pod \"f4a81084-193c-40f0-abda-964c909230c0\" (UID: \"f4a81084-193c-40f0-abda-964c909230c0\") " Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056319 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056423 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume" pod "f4a81084-193c-40f0-abda-964c909230c0" (UID: "f4a81084-193c-40f0-abda-964c909230c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056561 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056682 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czvcj\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056754 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056828 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.056934 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.057043 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.057096 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.057154 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.057314 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a81084-193c-40f0-abda-964c909230c0-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.058392 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs" pod "f4a81084-193c-40f0-abda-964c909230c0" (UID: "f4a81084-193c-40f0-abda-964c909230c0"). InnerVolumeSpecName "kube-api-access-5bsbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.162632 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume\") pod \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.162745 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp\" (UniqueName: \"kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp\") pod \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\" (UID: \"51bcca4d-52c3-4681-a8c9-f7e090ae567a\") " Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163016 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163081 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvcj\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163110 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163145 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163215 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163270 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163306 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163320 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163391 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163527 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bsbs\" (UniqueName: \"kubernetes.io/projected/f4a81084-193c-40f0-abda-964c909230c0-kube-api-access-5bsbs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.163890 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.164200 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume" pod "51bcca4d-52c3-4681-a8c9-f7e090ae567a" (UID: "51bcca4d-52c3-4681-a8c9-f7e090ae567a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.165119 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.165671 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.165904 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.166573 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp" pod "51bcca4d-52c3-4681-a8c9-f7e090ae567a" (UID: "51bcca4d-52c3-4681-a8c9-f7e090ae567a"). InnerVolumeSpecName "kube-api-access-tkqzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.166616 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.167301 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.168790 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.178419 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.179225 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvcj\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj\") pod \"e2e-api-a-a1deb6-openshift-default-744454789f-49dvt\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.265676 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bcca4d-52c3-4681-a8c9-f7e090ae567a-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.265702 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkqzp\" (UniqueName: \"kubernetes.io/projected/51bcca4d-52c3-4681-a8c9-f7e090ae567a-kube-api-access-tkqzp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.294837 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.571014 5043 generic.go:356] "Generic (PLEG): container finished" podID="51bcca4d-52c3-4681-a8c9-f7e090ae567a" containerID="25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43" exitCode=0 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.571184 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" event={"ID":"51bcca4d-52c3-4681-a8c9-f7e090ae567a","Type":"ContainerDied","Data":"25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.571240 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" event={"ID":"51bcca4d-52c3-4681-a8c9-f7e090ae567a","Type":"ContainerDied","Data":"0b65c63b27e1b6e4799c1f27095fcccf1c94256f2dc2abe7be4e638bfe2e6346"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.571271 5043 scope.go:122] "RemoveContainer" containerID="25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.571434 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.576407 5043 generic.go:356] "Generic (PLEG): container finished" podID="f4a81084-193c-40f0-abda-964c909230c0" containerID="c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6" exitCode=0 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.576581 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" event={"ID":"f4a81084-193c-40f0-abda-964c909230c0","Type":"ContainerDied","Data":"c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.576627 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.576634 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr" event={"ID":"f4a81084-193c-40f0-abda-964c909230c0","Type":"ContainerDied","Data":"bca25f4a4b8b2262458cced02e2625eeb2502175bb9b3869f17981153593e650"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.650172 5043 scope.go:122] "RemoveContainer" containerID="25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:07.650595 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43\": container with ID starting with 25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43 not found: ID does not exist" containerID="25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.650628 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43"} err="failed to get container status \"25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43\": rpc error: code = NotFound desc = could not find container \"25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43\": container with ID starting with 25d74ffbb6fc179fc20fac399cf8ad84f903274622e54c2215ed0251b9040c43 not found: ID does not exist" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.650653 5043 scope.go:122] "RemoveContainer" containerID="c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.667326 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.668787 5043 scope.go:122] "RemoveContainer" containerID="c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:07.669153 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6\": container with ID starting with c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6 not found: ID does not exist" containerID="c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.669190 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6"} err="failed to get container status \"c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6\": rpc error: code = NotFound desc = could not find container \"c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6\": container with ID starting with c05fdbc3933b00f1d0e64b3e199c23495ffa15ca3bc4c027076776cb118232f6 not found: ID does not exist" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.673107 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4eff5f-b5fddc59f-82pqr"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.683413 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.691353 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4eff5f-79b984d9f4-qsqvj"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:23:07.691524 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode703fb61_c4ed_4b10_95e0_cdec8d4b6c1d.slice/crio-8cbe78712160dda735fe6b2c019dd0eadc6527b4c974abddfbc86dc019866366 WatchSource:0}: Error finding container 8cbe78712160dda735fe6b2c019dd0eadc6527b4c974abddfbc86dc019866366: Status 404 returned error can't find the container with id 8cbe78712160dda735fe6b2c019dd0eadc6527b4c974abddfbc86dc019866366 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.695655 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.695720 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.695748 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.699218 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt"] Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.955139 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.958159 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51bcca4d-52c3-4681-a8c9-f7e090ae567a" path="/var/lib/kubelet/pods/51bcca4d-52c3-4681-a8c9-f7e090ae567a/volumes" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:07.959077 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a81084-193c-40f0-abda-964c909230c0" path="/var/lib/kubelet/pods/f4a81084-193c-40f0-abda-964c909230c0/volumes" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.568517 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.592583 5043 generic.go:356] "Generic (PLEG): container finished" podID="c769ec16-9843-4e45-9caf-298744983e65" containerID="78852d1ffea1e452bca13c755e516b7def266dd3b199c930c73723528e76076c" exitCode=0 Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.592676 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" event={"ID":"c769ec16-9843-4e45-9caf-298744983e65","Type":"ContainerDied","Data":"78852d1ffea1e452bca13c755e516b7def266dd3b199c930c73723528e76076c"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.593266 5043 scope.go:122] "RemoveContainer" containerID="78852d1ffea1e452bca13c755e516b7def266dd3b199c930c73723528e76076c" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.595315 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.595401 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.595450 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.601274 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" event={"ID":"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d","Type":"ContainerStarted","Data":"b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.601325 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" event={"ID":"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d","Type":"ContainerStarted","Data":"8cbe78712160dda735fe6b2c019dd0eadc6527b4c974abddfbc86dc019866366"} Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:08.693970 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podStartSLOduration=2.693949083 podStartE2EDuration="2.693949083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:08.662308353 +0000 UTC m=+2014.891862765" watchObservedRunningTime="2026-10-31 23:23:08.693949083 +0000 UTC m=+2014.923503497" Oct 31 23:23:09.166851 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:09.001277 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:23:09.295610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:09.295412 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:09.297107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:09.297074 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 23:23:09.297265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:09.297136 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 23:23:09.561252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:09.561065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:09.659434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:09.658356 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" event={"ID":"c769ec16-9843-4e45-9caf-298744983e65","Type":"ContainerStarted","Data":"e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9"} Oct 31 23:23:10.196934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.196901 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:23:10.197503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.197031 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:23:10.197503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.197098 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:23:10.296166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.296121 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 23:23:10.296369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.296208 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 23:23:10.703616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.697258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:10.843955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.843916 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:23:10.844256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.844227 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:10.906684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.906630 5043 generic.go:356] "Generic (PLEG): container finished" podID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerID="c6dffb4bc897d66cc9d638d13d4a567588ca65958fda26121828e98a37966319" exitCode=0 Oct 31 23:23:10.907911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.907876 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" event={"ID":"849be6ec-37ac-48a0-b865-844a8fcf8f10","Type":"ContainerDied","Data":"c6dffb4bc897d66cc9d638d13d4a567588ca65958fda26121828e98a37966319"} Oct 31 23:23:10.908055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.907982 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" event={"ID":"849be6ec-37ac-48a0-b865-844a8fcf8f10","Type":"ContainerDied","Data":"72f6bdbc3be59d179094c3ea76d569bc3ea0e253e26d0df800c77178c3c64967"} Oct 31 23:23:10.908055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.908013 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f6bdbc3be59d179094c3ea76d569bc3ea0e253e26d0df800c77178c3c64967" Oct 31 23:23:10.981641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.959943 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.982609 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.982786 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.982866 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.983079 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.983342 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.983815 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.983679 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.984041 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.984204 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.984439 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c\") pod \"849be6ec-37ac-48a0-b865-844a8fcf8f10\" (UID: \"849be6ec-37ac-48a0-b865-844a8fcf8f10\") " Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.984650 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:10.985796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.985112 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:10.986659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.985528 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.987800 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.987829 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.987847 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.987862 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849be6ec-37ac-48a0-b865-844a8fcf8f10-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.990867 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.994085 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.994931 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "kube-api-access-kqk9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:10.996906 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.997087 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:11.003611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:10.999768 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo" pod "849be6ec-37ac-48a0-b865-844a8fcf8f10" (UID: "849be6ec-37ac-48a0-b865-844a8fcf8f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:11.131656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.131582 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.131656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.131628 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.131656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.131643 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.131656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.131659 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849be6ec-37ac-48a0-b865-844a8fcf8f10-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.132037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.131675 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqk9c\" (UniqueName: \"kubernetes.io/projected/849be6ec-37ac-48a0-b865-844a8fcf8f10-kube-api-access-kqk9c\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:11.281569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.281501 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:23:11.282084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.281578 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:23:11.312606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.300339 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 23:23:11.312606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.300411 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 23:23:11.585631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.559022 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:11.807050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.807011 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx"] Oct 31 23:23:11.808823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.808792 5043 cpu_manager.go:414] "containerMap: removing container" podUID="51bcca4d-52c3-4681-a8c9-f7e090ae567a" containerName="payload-processing" Oct 31 23:23:11.809008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.808993 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51bcca4d-52c3-4681-a8c9-f7e090ae567a" containerName="payload-processing" Oct 31 23:23:11.809141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.809129 5043 cpu_manager.go:414] "containerMap: removing container" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" Oct 31 23:23:11.809254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.809241 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" Oct 31 23:23:11.809348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.809335 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f4a81084-193c-40f0-abda-964c909230c0" containerName="payload-pre-processing" Oct 31 23:23:11.809434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.809423 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a81084-193c-40f0-abda-964c909230c0" containerName="payload-pre-processing" Oct 31 23:23:11.811335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.811310 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="51bcca4d-52c3-4681-a8c9-f7e090ae567a" containerName="payload-processing" Oct 31 23:23:11.811509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.811497 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" containerName="istio-proxy" Oct 31 23:23:11.811619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.811608 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a81084-193c-40f0-abda-964c909230c0" containerName="payload-pre-processing" Oct 31 23:23:11.849444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.842243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:11.918844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.918813 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx"] Oct 31 23:23:11.930508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:11.924759 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.092481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.067869 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.103648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.094651 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.110674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.110623 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.116481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.110987 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.122394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.122352 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.124691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.124662 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ql6\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.124895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.124876 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.132508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.132470 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.138312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.132810 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.176757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.175511 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb" Oct 31 23:23:12.201066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.201030 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:23:12.201324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.201297 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:23:12.201778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.201749 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln"] Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277168 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277268 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277442 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277506 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277609 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277664 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ql6\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277700 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277725 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.277805 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.278525 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.280202 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.280463 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.280997 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.281674 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.288888 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln"] Oct 31 23:23:12.292791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.289028 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.295931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.295633 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 23:23:12.295931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.295696 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 23:23:12.302940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.302905 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.303444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.303422 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.304131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.304109 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ql6\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.306876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.306827 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.373762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.373590 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.379343 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.379485 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.379540 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.379899 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnz7\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.379990 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.380046 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.380147 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.380195 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.389172 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.380245 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.404683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.404643 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb"] Oct 31 23:23:12.423978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.423889 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-479107-openshift-default-6bfcbcf6cc-8m5hb"] Oct 31 23:23:12.492910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.492862 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.492973 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493133 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493200 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493405 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnz7\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493443 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493514 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493718 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493749 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.493959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.493807 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.494417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.494193 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.494417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.494315 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.495037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.495014 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.495201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.495180 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.497358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.496913 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.498395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.498369 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.507748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.507717 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnz7\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.511772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.511646 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token\") pod \"e2e-api-b-885fdf-openshift-default-758d98b657-k6vln\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:12.632168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.611085 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:12.737152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:12.736879 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:13.197174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.197136 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:23:13.197338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.197210 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:23:13.198075 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:23:13.198043 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1034845_c776_4a60_acb5_b94609230c97.slice/crio-eead1cb941e2cdb366ba9ca1d66b8274c6c91094fc7ed162dab2cbe702c93c2b WatchSource:0}: Error finding container eead1cb941e2cdb366ba9ca1d66b8274c6c91094fc7ed162dab2cbe702c93c2b: Status 404 returned error can't find the container with id eead1cb941e2cdb366ba9ca1d66b8274c6c91094fc7ed162dab2cbe702c93c2b Oct 31 23:23:13.203455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.202167 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:13.203455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.202257 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:13.203455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.202295 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:13.283567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.281854 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx"] Oct 31 23:23:13.297529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.297094 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 23:23:13.297529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.297153 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 23:23:13.595564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.587306 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:13.711862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.710928 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:13.711862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.711052 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:13.711862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:13.711099 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:14.125639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.113321 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849be6ec-37ac-48a0-b865-844a8fcf8f10" path="/var/lib/kubelet/pods/849be6ec-37ac-48a0-b865-844a8fcf8f10/volumes" Oct 31 23:23:14.125639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.113939 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln"] Oct 31 23:23:14.125639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.113995 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n"] Oct 31 23:23:14.125639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.114027 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285962ef-5bd6n"] Oct 31 23:23:14.204255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.204177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:14.204582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.204520 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:23:14.204721 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.204605 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:23:14.295705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.295185 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 23:23:14.295705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.295245 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 23:23:14.395568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.395385 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" event={"ID":"e1034845-c776-4a60-acb5-b94609230c97","Type":"ContainerStarted","Data":"df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb"} Oct 31 23:23:14.395568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.395456 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" event={"ID":"e1034845-c776-4a60-acb5-b94609230c97","Type":"ContainerStarted","Data":"eead1cb941e2cdb366ba9ca1d66b8274c6c91094fc7ed162dab2cbe702c93c2b"} Oct 31 23:23:14.465570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.465252 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" event={"ID":"b8a6417a-9af3-4344-86dc-099ba6da3cc8","Type":"ContainerStarted","Data":"8746e3c177007ffb31de2a37eb51b96d7246d6267b8f022221ef1cd409caa552"} Oct 31 23:23:14.627557 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.612021 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:14.683041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.682919 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podStartSLOduration=3.682896922 podStartE2EDuration="3.682896922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:14.452031103 +0000 UTC m=+2020.681585515" watchObservedRunningTime="2026-10-31 23:23:14.682896922 +0000 UTC m=+2020.912451337" Oct 31 23:23:14.775600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:14.771743 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:15.199699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.199656 5043 patch_prober.go:29] interesting pod/e2e-api-b-ba9088-openshift-default-6869694875-6klsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 31 23:23:15.200016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.199712 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 31 23:23:15.308780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.305269 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:15.308780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.307221 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:15.308780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.308193 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:23:15.498629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.411480 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:15.498629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.438538 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:15.498629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.438618 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:15.992304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:15.992261 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f"] Oct 31 23:23:16.025102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.025068 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f"] Oct 31 23:23:16.117316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.106410 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.203066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.202615 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.238246 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.238380 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.238473 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.298771 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.298939 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.298972 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mgk\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.299070 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.299119 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.341691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.299319 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.462033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.461934 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462295 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462339 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mgk\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462415 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462457 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462591 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462700 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462803 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.463475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.462907 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.486573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.471725 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.499580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.487697 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.507564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.505230 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.532567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.510140 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:16.532567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.510194 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:16.532567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.524623 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.532567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.525328 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:16.656676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.647499 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce21a44f-8c15-4e6e-9cfd-f01479405787" path="/var/lib/kubelet/pods/ce21a44f-8c15-4e6e-9cfd-f01479405787/volumes" Oct 31 23:23:16.698327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.698255 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" event={"ID":"b8a6417a-9af3-4344-86dc-099ba6da3cc8","Type":"ContainerStarted","Data":"c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c"} Oct 31 23:23:16.698653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.698625 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d"] Oct 31 23:23:16.698812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.698801 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s"] Oct 31 23:23:16.867020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.828613 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:23:16.867020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.847376 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:23:16.867020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.847672 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:16.895604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.874119 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:16.895604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.874185 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:16.938100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.938002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:17.013627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.992896 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:17.013627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.993322 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:23:17.013627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.995421 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podStartSLOduration=5.995403609 podStartE2EDuration="5.995403609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:16.248174022 +0000 UTC m=+2022.477728433" watchObservedRunningTime="2026-10-31 23:23:16.995403609 +0000 UTC m=+2023.224958021" Oct 31 23:23:17.013627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.997111 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mgk\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:17.013627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:16.997589 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:17.022442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.022369 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy\") pod \"e2e-api-b-836c99-openshift-default-8f564d59f-phs6f\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:17.202635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.165889 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:17.305074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.305003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:17.400171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.400138 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:17.400401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.400365 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:17.489148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.488900 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:17.846630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.757986 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:17.846630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:17.758044 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:18.292685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.291339 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl"] Oct 31 23:23:18.294666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.294600 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl"] Oct 31 23:23:18.387014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.385924 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:18.387014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.385975 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:18.496766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.496732 5043 generic.go:356] "Generic (PLEG): container finished" podID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerID="6b83c0f880f764c89884bd5c57afb6261257f1ebdc75e46acd9c162938987327" exitCode=0 Oct 31 23:23:18.498487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.497531 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" event={"ID":"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0","Type":"ContainerDied","Data":"6b83c0f880f764c89884bd5c57afb6261257f1ebdc75e46acd9c162938987327"} Oct 31 23:23:18.498487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.497990 5043 scope.go:122] "RemoveContainer" containerID="6b83c0f880f764c89884bd5c57afb6261257f1ebdc75e46acd9c162938987327" Oct 31 23:23:18.500578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.500409 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:18.500578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.500497 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:18.500578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.500565 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:18.604211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.604184 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f"] Oct 31 23:23:18.604650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.604528 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:18.604650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.604627 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:18.604820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.604669 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:18.659978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.657650 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:18.791992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.774441 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:18.791992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.774495 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:18.892320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.838853 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:23:18.892320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:18.838905 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:19.392593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.391484 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:19.392593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.391561 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:19.603415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.602167 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" event={"ID":"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0","Type":"ContainerStarted","Data":"48dcced70c7a43a399e0827691f5be6beb8a7434633296cbc8c2964b64eacce5"} Oct 31 23:23:19.739590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.691813 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" event={"ID":"c2797e68-0138-44e2-bc82-71ed4daa86b5","Type":"ContainerStarted","Data":"68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5"} Oct 31 23:23:19.739590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.691881 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" event={"ID":"c2797e68-0138-44e2-bc82-71ed4daa86b5","Type":"ContainerStarted","Data":"bfcb72e1386b5791af1c359ad70df36adba4fa07b0194042637a9dde2bc73ba2"} Oct 31 23:23:19.782571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.767305 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:19.782571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.767363 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:19.785587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.785497 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:19.874668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:19.874447 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podStartSLOduration=4.874425935 podStartE2EDuration="4.874425935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:19.873961247 +0000 UTC m=+2026.103515659" watchObservedRunningTime="2026-10-31 23:23:19.874425935 +0000 UTC m=+2026.103980347" Oct 31 23:23:20.049577 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:20.048428 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:23:20.135787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.135630 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:23:20.136106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.136080 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:20.136306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.136281 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:20.194631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.178211 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:20.194631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.178287 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:20.194631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.178618 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:20.559572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.511026 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:20.559910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.559865 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:20.786567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.785660 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:20.786567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:20.785727 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:21.158718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.084213 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:21.158718 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:21.100148 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:23:21.273636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.270272 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:21.277567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.273902 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:21.283477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.283432 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:21.283713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.283682 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:21.334081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.309795 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:21.672568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.596921 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:21.786566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.733971 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 23:23:21.786566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.734063 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 23:23:21.786566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.745507 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" podUID="6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" containerName="payload-processing" containerID="cri-o://33b03370dc51bb588203b26f74ca981d35efa15a889724c2f87a651d0a2a4cf9" gracePeriod=25 Oct 31 23:23:21.786566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.762583 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" podUID="bd32bc24-e23a-4a73-a804-00bb90c9c3e4" containerName="payload-pre-processing" containerID="cri-o://58613b10c1f9355d876a9a5effe6d9dd3f410db92370fd6cb303f31a3e2d7f64" gracePeriod=25 Oct 31 23:23:21.786566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.762809 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:21.988173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:21.988078 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:22.024773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.024721 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:22.025131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.025099 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:22.157408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.157371 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:22.159071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.159029 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:22.159907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.159877 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:22.325460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.218885 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:22.326246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.265624 5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 23:23:22.326951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.326016 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:22.334973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.334667 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:22.597103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.451646 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:22.718566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.716641 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:22.741573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.737665 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:22.823613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.817362 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 23:23:22.823613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.817445 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 23:23:22.823613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.818157 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:23:22.823613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.818237 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:23:22.823613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.818307 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:22.860966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.831996 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 23:23:22.911623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.889890 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:22.911623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.890002 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:23:22.949845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.949733 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:23:22.951713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:22.951334 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:23:23.211726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.199668 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:23.212391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.212356 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:23.219594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.201843 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:23.219813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.199691 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:23.220016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.219992 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:23.229652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.201634 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:23:23.250583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.243269 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:23:23.464240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.292679 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" podUID="79bf92b6-7040-4853-810a-87a4ce36f91d" containerName="payload-processing" containerID="cri-o://069b7c2398cd644886f7b4bc89af66b2b207b4d64b4cf992ca64a1c468885243" gracePeriod=25 Oct 31 23:23:23.486674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.297086 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" podUID="0d26d9e9-c130-429e-aa1d-234248fa63aa" containerName="payload-pre-processing" containerID="cri-o://feb06e08cb89c32e3daba35a09b2b12c8353632ad28f7289d0089d3da099bdc8" gracePeriod=25 Oct 31 23:23:23.504010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.501707 5043 generic.go:356] "Generic (PLEG): container finished" podID="6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" containerID="33b03370dc51bb588203b26f74ca981d35efa15a889724c2f87a651d0a2a4cf9" exitCode=0 Oct 31 23:23:23.504010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:23.503199 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" event={"ID":"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa","Type":"ContainerDied","Data":"33b03370dc51bb588203b26f74ca981d35efa15a889724c2f87a651d0a2a4cf9"} Oct 31 23:23:24.231714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.195756 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:24.231714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.195822 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:24.231714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.196248 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:24.231714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.196285 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:24.231714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.196798 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:24.498840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.498471 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:24.501476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.500221 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:24.505601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.502077 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:24.595574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.588680 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:23:24.708616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.706705 5043 generic.go:356] "Generic (PLEG): container finished" podID="bd32bc24-e23a-4a73-a804-00bb90c9c3e4" containerID="58613b10c1f9355d876a9a5effe6d9dd3f410db92370fd6cb303f31a3e2d7f64" exitCode=0 Oct 31 23:23:24.708616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:24.707075 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" event={"ID":"bd32bc24-e23a-4a73-a804-00bb90c9c3e4","Type":"ContainerDied","Data":"58613b10c1f9355d876a9a5effe6d9dd3f410db92370fd6cb303f31a3e2d7f64"} Oct 31 23:23:25.010246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.006208 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:25.177289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.169653 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:25.177289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.169715 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:25.182161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.182129 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:25.182404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.182373 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:25.244778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.244745 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:23:25.533863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.512136 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume\") pod \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " Oct 31 23:23:25.533863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.513003 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl\" (UniqueName: \"kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl\") pod \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\" (UID: \"bd32bc24-e23a-4a73-a804-00bb90c9c3e4\") " Oct 31 23:23:25.533863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.524064 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume" pod "bd32bc24-e23a-4a73-a804-00bb90c9c3e4" (UID: "bd32bc24-e23a-4a73-a804-00bb90c9c3e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:25.614982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.580786 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl" pod "bd32bc24-e23a-4a73-a804-00bb90c9c3e4" (UID: "bd32bc24-e23a-4a73-a804-00bb90c9c3e4"). InnerVolumeSpecName "kube-api-access-6zgrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:25.650569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.639281 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgrl\" (UniqueName: \"kubernetes.io/projected/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-kube-api-access-6zgrl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:25.650569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.639321 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd32bc24-e23a-4a73-a804-00bb90c9c3e4-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:25.695625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.690081 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:23:25.854663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.844222 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:25.923566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.921379 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl\" (UniqueName: \"kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl\") pod \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " Oct 31 23:23:25.923566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.921637 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume\") pod \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\" (UID: \"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa\") " Oct 31 23:23:25.979570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:25.969842 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume" pod "6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" (UID: "6dd2b67e-7cfc-4de8-951b-0d0aab7306fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:26.172624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.171257 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:26.172624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.171320 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:26.172624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.172080 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:26.172624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.172121 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:26.187336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.187296 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:26.192398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.192364 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl" pod "6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" (UID: "6dd2b67e-7cfc-4de8-951b-0d0aab7306fa"). InnerVolumeSpecName "kube-api-access-msxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:26.208332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.208299 5043 generic.go:356] "Generic (PLEG): container finished" podID="79bf92b6-7040-4853-810a-87a4ce36f91d" containerID="069b7c2398cd644886f7b4bc89af66b2b207b4d64b4cf992ca64a1c468885243" exitCode=0 Oct 31 23:23:26.208665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.208421 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" event={"ID":"79bf92b6-7040-4853-810a-87a4ce36f91d","Type":"ContainerDied","Data":"069b7c2398cd644886f7b4bc89af66b2b207b4d64b4cf992ca64a1c468885243"} Oct 31 23:23:26.250658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.246060 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" event={"ID":"bd32bc24-e23a-4a73-a804-00bb90c9c3e4","Type":"ContainerDied","Data":"4ef836bcf9ad0a3fd64f15e3f59d3f64ed01347b2298312836cc55fe969566ef"} Oct 31 23:23:26.250658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.246153 5043 scope.go:122] "RemoveContainer" containerID="58613b10c1f9355d876a9a5effe6d9dd3f410db92370fd6cb303f31a3e2d7f64" Oct 31 23:23:26.250658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.246396 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s" Oct 31 23:23:26.321419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.312609 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxrl\" (UniqueName: \"kubernetes.io/projected/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa-kube-api-access-msxrl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:26.370297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.369736 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" Oct 31 23:23:26.370297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.369738 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d" event={"ID":"6dd2b67e-7cfc-4de8-951b-0d0aab7306fa","Type":"ContainerDied","Data":"7ed4a4e1449fb7f4eb8ac799443f4586dae7b28585f08c351625da6d17c41658"} Oct 31 23:23:26.401234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.400491 5043 scope.go:122] "RemoveContainer" containerID="33b03370dc51bb588203b26f74ca981d35efa15a889724c2f87a651d0a2a4cf9" Oct 31 23:23:26.406126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.406094 5043 generic.go:356] "Generic (PLEG): container finished" podID="0d26d9e9-c130-429e-aa1d-234248fa63aa" containerID="feb06e08cb89c32e3daba35a09b2b12c8353632ad28f7289d0089d3da099bdc8" exitCode=0 Oct 31 23:23:26.406868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.406836 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" event={"ID":"0d26d9e9-c130-429e-aa1d-234248fa63aa","Type":"ContainerDied","Data":"feb06e08cb89c32e3daba35a09b2b12c8353632ad28f7289d0089d3da099bdc8"} Oct 31 23:23:26.511278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.511240 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s"] Oct 31 23:23:26.919102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.919065 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-885fdf-66bdd6b495-mj54s"] Oct 31 23:23:26.933452 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.933341 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d"] Oct 31 23:23:26.944905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:26.944834 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-885fdf-6fdc87d4fd-rdd7d"] Oct 31 23:23:27.074784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.074696 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr"] Oct 31 23:23:27.076422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.076392 5043 cpu_manager.go:414] "containerMap: removing container" podUID="bd32bc24-e23a-4a73-a804-00bb90c9c3e4" containerName="payload-pre-processing" Oct 31 23:23:27.076610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.076594 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd32bc24-e23a-4a73-a804-00bb90c9c3e4" containerName="payload-pre-processing" Oct 31 23:23:27.076757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.076744 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" containerName="payload-processing" Oct 31 23:23:27.076842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.076830 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" containerName="payload-processing" Oct 31 23:23:27.077236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.077158 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" containerName="payload-processing" Oct 31 23:23:27.077386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.077372 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd32bc24-e23a-4a73-a804-00bb90c9c3e4" containerName="payload-pre-processing" Oct 31 23:23:27.105728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.083771 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.143577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.086819 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr"] Oct 31 23:23:27.149660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.148883 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:27.174217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.174114 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:27.202572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.175322 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:27.281103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.189045 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:27.281103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.280444 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:27.285523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.285492 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.285808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.285782 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.286040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.286021 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.286216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.286198 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.286381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.286365 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.286511 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.286497 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.286676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.286659 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.286937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.286915 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.287114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.287081 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck6w\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.353368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.352912 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402120 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402266 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402342 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402410 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402458 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402486 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402687 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402733 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck6w\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.402959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.402861 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.403638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.403339 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.403698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.403661 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.403925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.403756 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.408406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.403923 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.408406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.404418 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.408927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.408905 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.409593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.409533 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.487731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.487682 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck6w\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.488507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.488477 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token\") pod \"e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.488859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.488839 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:23:27.490756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.490718 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" Oct 31 23:23:27.490878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.490756 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" event={"ID":"79bf92b6-7040-4853-810a-87a4ce36f91d","Type":"ContainerDied","Data":"bc6ede944a6852b154390ee3e099b076cc201d8ab051f7cd1600e932adb643e2"} Oct 31 23:23:27.490878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.490815 5043 scope.go:122] "RemoveContainer" containerID="069b7c2398cd644886f7b4bc89af66b2b207b4d64b4cf992ca64a1c468885243" Oct 31 23:23:27.496884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.496719 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" Oct 31 23:23:27.496884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.496738 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" event={"ID":"0d26d9e9-c130-429e-aa1d-234248fa63aa","Type":"ContainerDied","Data":"c4b212f0dad785cc1e99623b421e6c8f4536d802972773ae938fc0de6b7e98cc"} Oct 31 23:23:27.504737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.504715 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume\") pod \"79bf92b6-7040-4853-810a-87a4ce36f91d\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " Oct 31 23:23:27.504852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.504773 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m\" (UniqueName: \"kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m\") pod \"79bf92b6-7040-4853-810a-87a4ce36f91d\" (UID: \"79bf92b6-7040-4853-810a-87a4ce36f91d\") " Oct 31 23:23:27.505676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.505652 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume" pod "79bf92b6-7040-4853-810a-87a4ce36f91d" (UID: "79bf92b6-7040-4853-810a-87a4ce36f91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:27.507903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.507365 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m" pod "79bf92b6-7040-4853-810a-87a4ce36f91d" (UID: "79bf92b6-7040-4853-810a-87a4ce36f91d"). InnerVolumeSpecName "kube-api-access-q2c8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:27.507903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.507507 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bf92b6-7040-4853-810a-87a4ce36f91d-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:27.507903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.507528 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2c8m\" (UniqueName: \"kubernetes.io/projected/79bf92b6-7040-4853-810a-87a4ce36f91d-kube-api-access-q2c8m\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:27.507903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.507737 5043 scope.go:122] "RemoveContainer" containerID="feb06e08cb89c32e3daba35a09b2b12c8353632ad28f7289d0089d3da099bdc8" Oct 31 23:23:27.608461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.608388 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume\") pod \"0d26d9e9-c130-429e-aa1d-234248fa63aa\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " Oct 31 23:23:27.608632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.608593 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x\" (UniqueName: \"kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x\") pod \"0d26d9e9-c130-429e-aa1d-234248fa63aa\" (UID: \"0d26d9e9-c130-429e-aa1d-234248fa63aa\") " Oct 31 23:23:27.609653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.609629 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume" pod "0d26d9e9-c130-429e-aa1d-234248fa63aa" (UID: "0d26d9e9-c130-429e-aa1d-234248fa63aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:27.678580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.669393 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x" pod "0d26d9e9-c130-429e-aa1d-234248fa63aa" (UID: "0d26d9e9-c130-429e-aa1d-234248fa63aa"). InnerVolumeSpecName "kube-api-access-9bp9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:27.711667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.711632 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d26d9e9-c130-429e-aa1d-234248fa63aa-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:27.711824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.711670 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bp9x\" (UniqueName: \"kubernetes.io/projected/0d26d9e9-c130-429e-aa1d-234248fa63aa-kube-api-access-9bp9x\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:27.786591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.786557 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:27.889472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.889379 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl"] Oct 31 23:23:27.895763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.895729 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl"] Oct 31 23:23:27.901339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.901314 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl"] Oct 31 23:23:27.906942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:27.906913 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl"] Oct 31 23:23:28.004447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.004422 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d26d9e9-c130-429e-aa1d-234248fa63aa" path="/var/lib/kubelet/pods/0d26d9e9-c130-429e-aa1d-234248fa63aa/volumes" Oct 31 23:23:28.005282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.005260 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd2b67e-7cfc-4de8-951b-0d0aab7306fa" path="/var/lib/kubelet/pods/6dd2b67e-7cfc-4de8-951b-0d0aab7306fa/volumes" Oct 31 23:23:28.006064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.006043 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bf92b6-7040-4853-810a-87a4ce36f91d" path="/var/lib/kubelet/pods/79bf92b6-7040-4853-810a-87a4ce36f91d/volumes" Oct 31 23:23:28.007208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.007188 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd32bc24-e23a-4a73-a804-00bb90c9c3e4" path="/var/lib/kubelet/pods/bd32bc24-e23a-4a73-a804-00bb90c9c3e4/volumes" Oct 31 23:23:28.048512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.048332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:28.133615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.131948 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Oct 31 23:23:28.133615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.132006 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Oct 31 23:23:28.175163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.175073 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 23:23:28.175351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.175159 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 23:23:28.252711 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:23:28.252679 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4a82c_f1bf_4f69_9763_007d39edc7cd.slice/crio-a9ab8b4e4e7ba721a3490e72a08c30ce88fd370d03f36aea34dad29a9bfc718c WatchSource:0}: Error finding container a9ab8b4e4e7ba721a3490e72a08c30ce88fd370d03f36aea34dad29a9bfc718c: Status 404 returned error can't find the container with id a9ab8b4e4e7ba721a3490e72a08c30ce88fd370d03f36aea34dad29a9bfc718c Oct 31 23:23:28.262415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.261085 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr"] Oct 31 23:23:28.264128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.264089 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:28.264485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.264454 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:28.264615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.264538 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:28.311145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.310702 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-ba9088-6c7cdf6bb8-znkkl" podUID="79bf92b6-7040-4853-810a-87a4ce36f91d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.196:50052\" within 1s: context deadline exceeded" Oct 31 23:23:28.311145 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:23:28.311104 5043 logging.go:55] [core] [Channel #1670 SubChannel #1673]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.196:50052", ServerName: "10.132.0.196:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.196:50052: operation was canceled" Oct 31 23:23:28.313083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.311775 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ba9088-76bd59c795-8f6sl" podUID="0d26d9e9-c130-429e-aa1d-234248fa63aa" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.197:50052\" within 1s: context deadline exceeded" Oct 31 23:23:28.313083 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:23:28.311928 5043 logging.go:55] [core] [Channel #1671 SubChannel #1672]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.197:50052", ServerName: "10.132.0.197:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.197:50052: operation was canceled" Oct 31 23:23:28.522036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.521874 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:28.552107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.552059 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" event={"ID":"4ba4a82c-f1bf-4f69-9763-007d39edc7cd","Type":"ContainerStarted","Data":"1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d"} Oct 31 23:23:28.552311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.552135 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" event={"ID":"4ba4a82c-f1bf-4f69-9763-007d39edc7cd","Type":"ContainerStarted","Data":"a9ab8b4e4e7ba721a3490e72a08c30ce88fd370d03f36aea34dad29a9bfc718c"} Oct 31 23:23:28.605781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.598093 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podStartSLOduration=2.598072434 podStartE2EDuration="2.598072434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:28.587925635 +0000 UTC m=+2034.817480047" watchObservedRunningTime="2026-10-31 23:23:28.598072434 +0000 UTC m=+2034.827626845" Oct 31 23:23:28.786659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.786620 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:28.920444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:28.920409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:29.130960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.130817 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:23:29.133426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.133397 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:23:29.177890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.177858 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:29.178532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.178501 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:29.179095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.179072 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:23:29.720874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.717751 5043 generic.go:356] "Generic (PLEG): container finished" podID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerID="ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50" exitCode=0 Oct 31 23:23:29.728060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.728022 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" event={"ID":"a302ec33-70dd-42ae-baa1-aaa48d20a369","Type":"ContainerDied","Data":"ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50"} Oct 31 23:23:29.764702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.764666 5043 scope.go:122] "RemoveContainer" containerID="ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50" Oct 31 23:23:29.780864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.780774 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:29.781032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.780876 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:29.781032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.780926 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:23:29.791951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.791664 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 23:23:29.791951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:29.791713 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 23:23:30.296581 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:30.296508 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ee218_d3e7_4837_98f8_642a6266fdf9.slice\": RecentStats: unable to find data in memory cache]" Oct 31 23:23:30.355817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:30.355780 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:30.752653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:30.737659 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" event={"ID":"a302ec33-70dd-42ae-baa1-aaa48d20a369","Type":"ContainerStarted","Data":"0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc"} Oct 31 23:23:30.790933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:30.790884 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 23:23:30.793145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:30.792734 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 23:23:31.699571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:31.698848 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:23:31.699571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:31.699085 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 23:23:31.699571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:31.699150 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 23:23:31.705642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:31.701861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:31.804170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:31.804136 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 23:23:31.804785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:31.804739 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 23:23:32.697446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:32.697393 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 23:23:32.697673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:32.697491 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 23:23:32.726569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:32.725165 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:23:32.787055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:32.787022 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 23:23:32.787294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:32.787269 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 23:23:33.030569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:33.029704 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:33.697581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:33.697164 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 23:23:33.697581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:33.697226 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 23:23:33.786306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:33.786268 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 23:23:33.786495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:33.786322 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 23:23:34.003571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:34.001387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:34.697477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:34.697439 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 23:23:34.698304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:34.698250 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 23:23:34.763834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:34.756262 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:34.789570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:34.789225 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 23:23:34.789570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:34.789285 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 23:23:35.548596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:35.544126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:35.697728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:35.697694 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 31 23:23:35.698150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:35.697753 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 31 23:23:35.791384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:35.791340 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:35.791899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:35.791877 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:35.792826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:35.792804 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:23:36.302605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:36.301633 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:36.731650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:36.731508 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:23:36.733185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:36.733158 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:23:36.962570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:36.959276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:37.630566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:37.627302 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:38.526569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:38.523635 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:39.228398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:39.228365 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:39.767693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:39.767652 5043 scope.go:122] "RemoveContainer" containerID="25c797e23000023de4bba8307a9d50c245ae8ff0af3dc93404df320e81071ed7" Oct 31 23:23:39.782776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:39.782753 5043 scope.go:122] "RemoveContainer" containerID="a60a1294668bd07f0b4a9b059700cc3f5f5797556cf6765066b3ece5bdd064b2" Oct 31 23:23:39.917899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:39.917864 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw"] Oct 31 23:23:39.969642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:39.969608 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c"] Oct 31 23:23:40.064738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:40.064650 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:40.217924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:40.214520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:40.567884 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:40.566893 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:23:41.020686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:41.020504 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:41.454854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:41.454765 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq"] Oct 31 23:23:41.455379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:41.455317 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" podUID="b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" containerName="maas-api" containerID="cri-o://4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99" gracePeriod=30 Oct 31 23:23:41.554258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:41.554218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:41.821923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:41.821871 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:42.079637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.079200 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:23:42.082184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.082156 5043 generic.go:356] "Generic (PLEG): container finished" podID="b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" containerID="4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99" exitCode=0 Oct 31 23:23:42.082314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.082228 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" event={"ID":"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e","Type":"ContainerDied","Data":"4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99"} Oct 31 23:23:42.082314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.082278 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" event={"ID":"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e","Type":"ContainerDied","Data":"4f779a711d0c12e97b4424a81694a61b8774eef063d2a1d0e85a6904a6097a73"} Oct 31 23:23:42.082314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.082301 5043 scope.go:122] "RemoveContainer" containerID="4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99" Oct 31 23:23:42.094279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.094259 5043 scope.go:122] "RemoveContainer" containerID="4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99" Oct 31 23:23:42.094538 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:42.094519 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99\": container with ID starting with 4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99 not found: ID does not exist" containerID="4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99" Oct 31 23:23:42.094631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.094562 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99"} err="failed to get container status \"4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99\": rpc error: code = NotFound desc = could not find container \"4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99\": container with ID starting with 4bf1760b8af845784d49f24eca855471020842a87f981ea5139afb4f8eb34a99 not found: ID does not exist" Oct 31 23:23:42.155150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.155110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:42.257457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.257411 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs\") pod \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " Oct 31 23:23:42.257738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.257477 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca\") pod \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " Oct 31 23:23:42.257738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.257523 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\") pod \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " Oct 31 23:23:42.258016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.257854 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m\" (UniqueName: \"kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m\") pod \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\" (UID: \"b8668771-c2e1-46e5-b3c2-d5fb0f57a91e\") " Oct 31 23:23:42.259523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.259496 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca" pod "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" (UID: "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:42.264879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.264834 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls" pod "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" (UID: "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:42.271590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.269092 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m" pod "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" (UID: "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e"). InnerVolumeSpecName "kube-api-access-45g2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:42.271590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.271425 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs" pod "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" (UID: "b8668771-c2e1-46e5-b3c2-d5fb0f57a91e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:42.361264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.361181 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45g2m\" (UniqueName: \"kubernetes.io/projected/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-kube-api-access-45g2m\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:42.361264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.361215 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:42.361264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.361230 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:42.361264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.361240 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:42.452372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.452332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:42.697397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:42.697307 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:43.092321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.092292 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq" Oct 31 23:23:43.131333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.124528 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq"] Oct 31 23:23:43.133034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.133001 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4eff5f-5644cb6bd9-2bgvq"] Oct 31 23:23:43.287187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.287145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:43.558169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.558125 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b"] Oct 31 23:23:43.662572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.660258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:43.812276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.812179 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:43.991570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:43.990217 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" path="/var/lib/kubelet/pods/b8668771-c2e1-46e5-b3c2-d5fb0f57a91e/volumes" Oct 31 23:23:44.407828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:44.407672 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7"] Oct 31 23:23:44.409849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:44.408735 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" podUID="00a8347c-472c-4b00-984d-e7b4739b6faa" containerName="maas-api" containerID="cri-o://d48f1afae4c7c4e5d02315dcf4c3103506cade7a97737749f0119cf6c6c9ab1c" gracePeriod=30 Oct 31 23:23:44.498905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:44.498871 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:44.814627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:44.814325 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:44.930571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:44.924348 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" podUID="f1a540c7-f029-4183-80f3-997c9cf3e3a7" containerName="payload-processing" containerID="cri-o://25b9be2fd3fdc3e9cffb55ee4182708450237f5e899d92260e1788270052d106" gracePeriod=25 Oct 31 23:23:45.038806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.005228 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" podUID="aa780837-0bd4-4676-88be-94ebb229cf22" containerName="payload-pre-processing" containerID="cri-o://40a8669644bc1096b91076162178f825184dc194d93146e6c17a306b8c269de8" gracePeriod=25 Oct 31 23:23:45.128477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.124569 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:45.271887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.271837 5043 generic.go:356] "Generic (PLEG): container finished" podID="aa780837-0bd4-4676-88be-94ebb229cf22" containerID="40a8669644bc1096b91076162178f825184dc194d93146e6c17a306b8c269de8" exitCode=0 Oct 31 23:23:45.272118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.272100 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" event={"ID":"aa780837-0bd4-4676-88be-94ebb229cf22","Type":"ContainerDied","Data":"40a8669644bc1096b91076162178f825184dc194d93146e6c17a306b8c269de8"} Oct 31 23:23:45.274758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.274727 5043 generic.go:356] "Generic (PLEG): container finished" podID="f1a540c7-f029-4183-80f3-997c9cf3e3a7" containerID="25b9be2fd3fdc3e9cffb55ee4182708450237f5e899d92260e1788270052d106" exitCode=0 Oct 31 23:23:45.275027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.274830 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" event={"ID":"f1a540c7-f029-4183-80f3-997c9cf3e3a7","Type":"ContainerDied","Data":"25b9be2fd3fdc3e9cffb55ee4182708450237f5e899d92260e1788270052d106"} Oct 31 23:23:45.276744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.276715 5043 generic.go:356] "Generic (PLEG): container finished" podID="00a8347c-472c-4b00-984d-e7b4739b6faa" containerID="d48f1afae4c7c4e5d02315dcf4c3103506cade7a97737749f0119cf6c6c9ab1c" exitCode=0 Oct 31 23:23:45.276910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.276890 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" event={"ID":"00a8347c-472c-4b00-984d-e7b4739b6faa","Type":"ContainerDied","Data":"d48f1afae4c7c4e5d02315dcf4c3103506cade7a97737749f0119cf6c6c9ab1c"} Oct 31 23:23:45.276988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.276938 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" event={"ID":"00a8347c-472c-4b00-984d-e7b4739b6faa","Type":"ContainerDied","Data":"f556d7df3be3479deb0259644fddfc43762d42cb1da3c6f67e3f6d49edb45fc3"} Oct 31 23:23:45.276988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.276962 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f556d7df3be3479deb0259644fddfc43762d42cb1da3c6f67e3f6d49edb45fc3" Oct 31 23:23:45.277296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.277274 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:23:45.288468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.288439 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8\" (UniqueName: \"kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8\") pod \"00a8347c-472c-4b00-984d-e7b4739b6faa\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " Oct 31 23:23:45.288614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.288586 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls\") pod \"00a8347c-472c-4b00-984d-e7b4739b6faa\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " Oct 31 23:23:45.288873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.288854 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs\") pod \"00a8347c-472c-4b00-984d-e7b4739b6faa\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " Oct 31 23:23:45.288947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.288932 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca\") pod \"00a8347c-472c-4b00-984d-e7b4739b6faa\" (UID: \"00a8347c-472c-4b00-984d-e7b4739b6faa\") " Oct 31 23:23:45.290113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.290071 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca" pod "00a8347c-472c-4b00-984d-e7b4739b6faa" (UID: "00a8347c-472c-4b00-984d-e7b4739b6faa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:45.291671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.291593 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:45.299497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.293820 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls" pod "00a8347c-472c-4b00-984d-e7b4739b6faa" (UID: "00a8347c-472c-4b00-984d-e7b4739b6faa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:45.301103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.300746 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8" pod "00a8347c-472c-4b00-984d-e7b4739b6faa" (UID: "00a8347c-472c-4b00-984d-e7b4739b6faa"). InnerVolumeSpecName "kube-api-access-668r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:45.344167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.340752 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs" pod "00a8347c-472c-4b00-984d-e7b4739b6faa" (UID: "00a8347c-472c-4b00-984d-e7b4739b6faa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:45.402129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.392531 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.402129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.392577 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a8347c-472c-4b00-984d-e7b4739b6faa-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.402129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.392593 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a8347c-472c-4b00-984d-e7b4739b6faa-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.402129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.392602 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668r8\" (UniqueName: \"kubernetes.io/projected/00a8347c-472c-4b00-984d-e7b4739b6faa-kube-api-access-668r8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.497616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.497527 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:45.586705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.586285 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:23:45.699213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.699119 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume\") pod \"aa780837-0bd4-4676-88be-94ebb229cf22\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " Oct 31 23:23:45.699410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.699388 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8\" (UniqueName: \"kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8\") pod \"aa780837-0bd4-4676-88be-94ebb229cf22\" (UID: \"aa780837-0bd4-4676-88be-94ebb229cf22\") " Oct 31 23:23:45.699596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.699575 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume" pod "aa780837-0bd4-4676-88be-94ebb229cf22" (UID: "aa780837-0bd4-4676-88be-94ebb229cf22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:45.700891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.700866 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa780837-0bd4-4676-88be-94ebb229cf22-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.704525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.704494 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8" pod "aa780837-0bd4-4676-88be-94ebb229cf22" (UID: "aa780837-0bd4-4676-88be-94ebb229cf22"). InnerVolumeSpecName "kube-api-access-czqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:45.783504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.783477 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:23:45.801406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.801371 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6\" (UniqueName: \"kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6\") pod \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " Oct 31 23:23:45.801723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.801706 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume\") pod \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\" (UID: \"f1a540c7-f029-4183-80f3-997c9cf3e3a7\") " Oct 31 23:23:45.802116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.802091 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume" pod "f1a540c7-f029-4183-80f3-997c9cf3e3a7" (UID: "f1a540c7-f029-4183-80f3-997c9cf3e3a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:45.802405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.802388 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a540c7-f029-4183-80f3-997c9cf3e3a7-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.802464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.802412 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czqh8\" (UniqueName: \"kubernetes.io/projected/aa780837-0bd4-4676-88be-94ebb229cf22-kube-api-access-czqh8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:45.803634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.803609 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6" pod "f1a540c7-f029-4183-80f3-997c9cf3e3a7" (UID: "f1a540c7-f029-4183-80f3-997c9cf3e3a7"). InnerVolumeSpecName "kube-api-access-jdbt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:45.904303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:45.904263 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbt6\" (UniqueName: \"kubernetes.io/projected/f1a540c7-f029-4183-80f3-997c9cf3e3a7-kube-api-access-jdbt6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:46.288573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.288514 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" Oct 31 23:23:46.288765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.288596 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw" event={"ID":"f1a540c7-f029-4183-80f3-997c9cf3e3a7","Type":"ContainerDied","Data":"0677514eb0c5ae19b4c76e530759cfee54aa397f4750d2875c9823908c61bc01"} Oct 31 23:23:46.288765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.288666 5043 scope.go:122] "RemoveContainer" containerID="25b9be2fd3fdc3e9cffb55ee4182708450237f5e899d92260e1788270052d106" Oct 31 23:23:46.290525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.290507 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" Oct 31 23:23:46.290699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.290565 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c" event={"ID":"aa780837-0bd4-4676-88be-94ebb229cf22","Type":"ContainerDied","Data":"7eb44700246657c11650991c06b14aae1bea099d30f92fb79d3f191e2b482ca0"} Oct 31 23:23:46.290781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.290757 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7" Oct 31 23:23:46.303013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.302998 5043 scope.go:122] "RemoveContainer" containerID="40a8669644bc1096b91076162178f825184dc194d93146e6c17a306b8c269de8" Oct 31 23:23:46.358943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.351761 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:46.384699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.384638 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw"] Oct 31 23:23:46.390276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.390247 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a1deb6-55d64dc5b-gbwvw"] Oct 31 23:23:46.402405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.402375 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7"] Oct 31 23:23:46.408418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.408388 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-885fdf-5949575cf9-v9mq7"] Oct 31 23:23:46.428292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.428252 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c"] Oct 31 23:23:46.437378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:46.437342 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a1deb6-8456f54767-49r6c"] Oct 31 23:23:47.237604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:47.235291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:47.893646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:47.893606 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:47.952939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:47.952898 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a8347c-472c-4b00-984d-e7b4739b6faa" path="/var/lib/kubelet/pods/00a8347c-472c-4b00-984d-e7b4739b6faa/volumes" Oct 31 23:23:47.953701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:47.953677 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa780837-0bd4-4676-88be-94ebb229cf22" path="/var/lib/kubelet/pods/aa780837-0bd4-4676-88be-94ebb229cf22/volumes" Oct 31 23:23:47.954281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:47.954259 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a540c7-f029-4183-80f3-997c9cf3e3a7" path="/var/lib/kubelet/pods/f1a540c7-f029-4183-80f3-997c9cf3e3a7/volumes" Oct 31 23:23:48.526490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:48.526454 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:48.560583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:48.558390 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" containerID="cri-o://b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b" gracePeriod=25 Oct 31 23:23:49.026128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.026098 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:23:49.090236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.090208 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume\") pod \"947422e9-fc75-4947-b557-cd04e8f24974\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " Oct 31 23:23:49.090401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.090330 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642\" (UniqueName: \"kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642\") pod \"947422e9-fc75-4947-b557-cd04e8f24974\" (UID: \"947422e9-fc75-4947-b557-cd04e8f24974\") " Oct 31 23:23:49.091121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.091090 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume" pod "947422e9-fc75-4947-b557-cd04e8f24974" (UID: "947422e9-fc75-4947-b557-cd04e8f24974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:49.092847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.092817 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642" pod "947422e9-fc75-4947-b557-cd04e8f24974" (UID: "947422e9-fc75-4947-b557-cd04e8f24974"). InnerVolumeSpecName "kube-api-access-qs642". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:49.192489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.192413 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs642\" (UniqueName: \"kubernetes.io/projected/947422e9-fc75-4947-b557-cd04e8f24974-kube-api-access-qs642\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:49.192489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.192444 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/947422e9-fc75-4947-b557-cd04e8f24974-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:49.346567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.338148 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:49.358941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.358842 5043 generic.go:356] "Generic (PLEG): container finished" podID="947422e9-fc75-4947-b557-cd04e8f24974" containerID="b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b" exitCode=0 Oct 31 23:23:49.359162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.359146 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" event={"ID":"947422e9-fc75-4947-b557-cd04e8f24974","Type":"ContainerDied","Data":"b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b"} Oct 31 23:23:49.359238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.359206 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" event={"ID":"947422e9-fc75-4947-b557-cd04e8f24974","Type":"ContainerDied","Data":"eb77938699c5584367aa542c0b089c75c467803599d7289e34b2ef6801d41f5f"} Oct 31 23:23:49.359298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.359241 5043 scope.go:122] "RemoveContainer" containerID="b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b" Oct 31 23:23:49.359351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.359326 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b" Oct 31 23:23:49.378666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.378637 5043 scope.go:122] "RemoveContainer" containerID="b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b" Oct 31 23:23:49.379168 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:49.379136 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b\": container with ID starting with b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b not found: ID does not exist" containerID="b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b" Oct 31 23:23:49.379284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.379176 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b"} err="failed to get container status \"b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b\": rpc error: code = NotFound desc = could not find container \"b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b\": container with ID starting with b139339922fcba25b119729dc24bb19377420f56ad84362b6ad8fab35e79567b not found: ID does not exist" Oct 31 23:23:49.424868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.424824 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b"] Oct 31 23:23:49.440658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.440626 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8e6a08-f65958dcf-mjq9b"] Oct 31 23:23:49.620422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.620380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:49.717788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.717747 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds"] Oct 31 23:23:49.718030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.718003 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" containerID="cri-o://7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04" gracePeriod=30 Oct 31 23:23:49.798274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.798236 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:49.980686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:49.966682 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947422e9-fc75-4947-b557-cd04e8f24974" path="/var/lib/kubelet/pods/947422e9-fc75-4947-b557-cd04e8f24974/volumes" Oct 31 23:23:50.345567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.344711 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd"] Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.345957 5043 cpu_manager.go:414] "containerMap: removing container" podUID="79bf92b6-7040-4853-810a-87a4ce36f91d" containerName="payload-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.345976 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bf92b6-7040-4853-810a-87a4ce36f91d" containerName="payload-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.345999 5043 cpu_manager.go:414] "containerMap: removing container" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346008 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346051 5043 cpu_manager.go:414] "containerMap: removing container" podUID="00a8347c-472c-4b00-984d-e7b4739b6faa" containerName="maas-api" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346058 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a8347c-472c-4b00-984d-e7b4739b6faa" containerName="maas-api" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346084 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" containerName="maas-api" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346091 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" containerName="maas-api" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346106 5043 cpu_manager.go:414] "containerMap: removing container" podUID="aa780837-0bd4-4676-88be-94ebb229cf22" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346114 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa780837-0bd4-4676-88be-94ebb229cf22" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346144 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f1a540c7-f029-4183-80f3-997c9cf3e3a7" containerName="payload-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346151 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a540c7-f029-4183-80f3-997c9cf3e3a7" containerName="payload-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346170 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0d26d9e9-c130-429e-aa1d-234248fa63aa" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346177 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d26d9e9-c130-429e-aa1d-234248fa63aa" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346332 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8668771-c2e1-46e5-b3c2-d5fb0f57a91e" containerName="maas-api" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346346 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa780837-0bd4-4676-88be-94ebb229cf22" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346362 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a540c7-f029-4183-80f3-997c9cf3e3a7" containerName="payload-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346373 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d26d9e9-c130-429e-aa1d-234248fa63aa" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346382 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bf92b6-7040-4853-810a-87a4ce36f91d" containerName="payload-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346390 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="947422e9-fc75-4947-b557-cd04e8f24974" containerName="payload-pre-processing" Oct 31 23:23:50.349587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.346404 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a8347c-472c-4b00-984d-e7b4739b6faa" containerName="maas-api" Oct 31 23:23:50.389579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.388278 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.484010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.482500 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.484010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.482620 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8cq\" (UniqueName: \"kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.484010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.482781 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.484010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.483522 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:23:50.516423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.492650 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd"] Oct 31 23:23:50.516423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.502948 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:23:50.588571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.588196 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.588571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.588292 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8cq\" (UniqueName: \"kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.588571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.588428 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.603177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.602874 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.604064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.603916 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.605448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.604785 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8cq\" (UniqueName: \"kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq\") pod \"maas-api-revoke-keys-e2e-api-b-836c99-n4hjd\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:50.826170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:50.826124 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:51.012952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:51.012904 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:51.231797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:51.217144 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd"] Oct 31 23:23:51.359565 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:23:51.357693 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad03db6_9f68_4499_abea_aa85bc145b09.slice/crio-9a9977f2a2d41837bfbc32fbb888bd823618526edd36a5cb98509379803dd440 WatchSource:0}: Error finding container 9a9977f2a2d41837bfbc32fbb888bd823618526edd36a5cb98509379803dd440: Status 404 returned error can't find the container with id 9a9977f2a2d41837bfbc32fbb888bd823618526edd36a5cb98509379803dd440 Oct 31 23:23:51.573563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:51.573509 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" event={"ID":"5ad03db6-9f68-4499-abea-aa85bc145b09","Type":"ContainerStarted","Data":"9a9977f2a2d41837bfbc32fbb888bd823618526edd36a5cb98509379803dd440"} Oct 31 23:23:52.225643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:52.221730 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:52.318567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:52.304781 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 23:23:52.589339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:52.589295 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" event={"ID":"5ad03db6-9f68-4499-abea-aa85bc145b09","Type":"ContainerStarted","Data":"5a445c07d73b4b812a2f53a8e7194a751b4a95531942adccee43e62a48479ebc"} Oct 31 23:23:52.613512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:52.613438 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" podStartSLOduration=2.613419893 podStartE2EDuration="2.613419893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:23:52.612241165 +0000 UTC m=+2058.841795577" watchObservedRunningTime="2026-10-31 23:23:52.613419893 +0000 UTC m=+2058.842974303" Oct 31 23:23:52.824936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:52.824847 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:53.601628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.601595 5043 generic.go:356] "Generic (PLEG): container finished" podID="5ad03db6-9f68-4499-abea-aa85bc145b09" containerID="5a445c07d73b4b812a2f53a8e7194a751b4a95531942adccee43e62a48479ebc" exitCode=0 Oct 31 23:23:53.602123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.601679 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" event={"ID":"5ad03db6-9f68-4499-abea-aa85bc145b09","Type":"ContainerDied","Data":"5a445c07d73b4b812a2f53a8e7194a751b4a95531942adccee43e62a48479ebc"} Oct 31 23:23:53.670160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.670117 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:53.859293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.859171 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:53.905021 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.904832 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl"] Oct 31 23:23:53.911271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.905079 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" containerID="cri-o://725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15" gracePeriod=30 Oct 31 23:23:53.912458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.912428 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm"] Oct 31 23:23:53.912689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.912658 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" containerID="cri-o://ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645" gracePeriod=30 Oct 31 23:23:53.923805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.923773 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln"] Oct 31 23:23:53.924022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:53.923991 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" containerID="cri-o://c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c" gracePeriod=30 Oct 31 23:23:54.067579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.064517 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:54.428365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.428322 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:23:54.428560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.428391 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:23:54.501962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.501924 5043 patch_prober.go:29] interesting pod/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:23:54.502159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.501989 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:23:54.864171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.864144 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:54.909043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:54.907968 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:55.056883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.056832 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca\") pod \"5ad03db6-9f68-4499-abea-aa85bc145b09\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " Oct 31 23:23:55.057162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.057139 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca\") pod \"5ad03db6-9f68-4499-abea-aa85bc145b09\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " Oct 31 23:23:55.057388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.057369 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq\" (UniqueName: \"kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq\") pod \"5ad03db6-9f68-4499-abea-aa85bc145b09\" (UID: \"5ad03db6-9f68-4499-abea-aa85bc145b09\") " Oct 31 23:23:55.059494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.059372 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca" pod "5ad03db6-9f68-4499-abea-aa85bc145b09" (UID: "5ad03db6-9f68-4499-abea-aa85bc145b09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:55.073281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.073253 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca" pod "5ad03db6-9f68-4499-abea-aa85bc145b09" (UID: "5ad03db6-9f68-4499-abea-aa85bc145b09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:55.074991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.074951 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq" pod "5ad03db6-9f68-4499-abea-aa85bc145b09" (UID: "5ad03db6-9f68-4499-abea-aa85bc145b09"). InnerVolumeSpecName "kube-api-access-ng8cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:55.091761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.091741 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:23:55.161481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.161422 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.161481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.161471 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad03db6-9f68-4499-abea-aa85bc145b09-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.161481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.161489 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8cq\" (UniqueName: \"kubernetes.io/projected/5ad03db6-9f68-4499-abea-aa85bc145b09-kube-api-access-ng8cq\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.264834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.264800 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.264848 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.264895 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.264960 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.265081 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.265163 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.265282 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.265413 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.265462 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh\") pod \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\" (UID: \"a231fc23-2c9b-4337-bc59-fd95964c2a2f\") " Oct 31 23:23:55.265966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.265698 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:55.266409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.266380 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.266504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.266478 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:55.266633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.266577 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:55.266707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.266630 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:55.266707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.266662 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:55.268943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.268904 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:55.269036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.268948 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:55.269262 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.269231 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "kube-api-access-gpqzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:55.269401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.269380 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo" pod "a231fc23-2c9b-4337-bc59-fd95964c2a2f" (UID: "a231fc23-2c9b-4337-bc59-fd95964c2a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:55.367793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367762 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.367793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367789 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.367793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367799 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.368092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367807 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.368092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367816 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqzh\" (UniqueName: \"kubernetes.io/projected/a231fc23-2c9b-4337-bc59-fd95964c2a2f-kube-api-access-gpqzh\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.368092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367826 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.368092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367835 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a231fc23-2c9b-4337-bc59-fd95964c2a2f-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.368092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.367844 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a231fc23-2c9b-4337-bc59-fd95964c2a2f-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:55.668623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.665016 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" Oct 31 23:23:55.668623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.665045 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd" event={"ID":"5ad03db6-9f68-4499-abea-aa85bc145b09","Type":"ContainerDied","Data":"9a9977f2a2d41837bfbc32fbb888bd823618526edd36a5cb98509379803dd440"} Oct 31 23:23:55.668623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.665111 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a9977f2a2d41837bfbc32fbb888bd823618526edd36a5cb98509379803dd440" Oct 31 23:23:55.674610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.674518 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:55.675168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.675139 5043 generic.go:356] "Generic (PLEG): container finished" podID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerID="7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04" exitCode=0 Oct 31 23:23:55.675412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.675192 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" Oct 31 23:23:55.676739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.675207 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" event={"ID":"a231fc23-2c9b-4337-bc59-fd95964c2a2f","Type":"ContainerDied","Data":"7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04"} Oct 31 23:23:55.676739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.675821 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds" event={"ID":"a231fc23-2c9b-4337-bc59-fd95964c2a2f","Type":"ContainerDied","Data":"45cc7139f153004b0bb732d5c5b3ca4e2c2e42f5cce86f6aca29d1e8f83dcc0e"} Oct 31 23:23:55.676739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.675857 5043 scope.go:122] "RemoveContainer" containerID="7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04" Oct 31 23:23:55.695361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.695339 5043 scope.go:122] "RemoveContainer" containerID="7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04" Oct 31 23:23:55.695742 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:55.695716 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04\": container with ID starting with 7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04 not found: ID does not exist" containerID="7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04" Oct 31 23:23:55.695855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.695755 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04"} err="failed to get container status \"7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04\": rpc error: code = NotFound desc = could not find container \"7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04\": container with ID starting with 7819da0f6764f74dff71b7920d2384730825487d309eefbb024a7e13f76dac04 not found: ID does not exist" Oct 31 23:23:55.710095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.710066 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds"] Oct 31 23:23:55.718888 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.718858 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4eff5f-openshift-default-857dd66bb5-6s6ds"] Oct 31 23:23:55.939043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:55.938956 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" path="/var/lib/kubelet/pods/a231fc23-2c9b-4337-bc59-fd95964c2a2f/volumes" Oct 31 23:23:56.287412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:56.287358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:56.758026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:56.757986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:56.812431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:56.812399 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9"] Oct 31 23:23:56.812665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:56.812639 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" podUID="5f9cb063-6cf6-4461-87af-5d1e48affa27" containerName="maas-api" containerID="cri-o://2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8" gracePeriod=30 Oct 31 23:23:56.911562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:56.911507 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:57.183628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.173145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:57.393915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.393891 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:23:57.462144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.462058 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:57.572805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.570270 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca\") pod \"5f9cb063-6cf6-4461-87af-5d1e48affa27\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " Oct 31 23:23:57.572805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.570363 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs\") pod \"5f9cb063-6cf6-4461-87af-5d1e48affa27\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " Oct 31 23:23:57.572805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.570507 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls\") pod \"5f9cb063-6cf6-4461-87af-5d1e48affa27\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " Oct 31 23:23:57.572805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.570600 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q\" (UniqueName: \"kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q\") pod \"5f9cb063-6cf6-4461-87af-5d1e48affa27\" (UID: \"5f9cb063-6cf6-4461-87af-5d1e48affa27\") " Oct 31 23:23:57.574832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.573439 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca" pod "5f9cb063-6cf6-4461-87af-5d1e48affa27" (UID: "5f9cb063-6cf6-4461-87af-5d1e48affa27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:57.582249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.582194 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls" pod "5f9cb063-6cf6-4461-87af-5d1e48affa27" (UID: "5f9cb063-6cf6-4461-87af-5d1e48affa27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:57.582393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.582305 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q" pod "5f9cb063-6cf6-4461-87af-5d1e48affa27" (UID: "5f9cb063-6cf6-4461-87af-5d1e48affa27"). InnerVolumeSpecName "kube-api-access-vrc9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:57.589353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.586621 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs" pod "5f9cb063-6cf6-4461-87af-5d1e48affa27" (UID: "5f9cb063-6cf6-4461-87af-5d1e48affa27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:23:57.615595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.608282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:57.675278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.675246 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9cb063-6cf6-4461-87af-5d1e48affa27-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.675278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.675275 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.675278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.675285 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9cb063-6cf6-4461-87af-5d1e48affa27-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.675509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.675299 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc9q\" (UniqueName: \"kubernetes.io/projected/5f9cb063-6cf6-4461-87af-5d1e48affa27-kube-api-access-vrc9q\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:57.703058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.703026 5043 generic.go:356] "Generic (PLEG): container finished" podID="5f9cb063-6cf6-4461-87af-5d1e48affa27" containerID="2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8" exitCode=0 Oct 31 23:23:57.703248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.703076 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" Oct 31 23:23:57.703248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.703113 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" event={"ID":"5f9cb063-6cf6-4461-87af-5d1e48affa27","Type":"ContainerDied","Data":"2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8"} Oct 31 23:23:57.703248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.703180 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9" event={"ID":"5f9cb063-6cf6-4461-87af-5d1e48affa27","Type":"ContainerDied","Data":"b8d64b97d8c5bd1f94d85c36c123c423acc85de30380625d186677cd96eb1b39"} Oct 31 23:23:57.703248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.703210 5043 scope.go:122] "RemoveContainer" containerID="2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8" Oct 31 23:23:57.715811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.715786 5043 scope.go:122] "RemoveContainer" containerID="2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8" Oct 31 23:23:57.716092 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:57.716071 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8\": container with ID starting with 2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8 not found: ID does not exist" containerID="2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8" Oct 31 23:23:57.716163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.716100 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8"} err="failed to get container status \"2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8\": rpc error: code = NotFound desc = could not find container \"2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8\": container with ID starting with 2395a6dfa55047aa65c11e2416873b279074619f89d14472c5109782abdbccf8 not found: ID does not exist" Oct 31 23:23:57.750250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.750219 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9"] Oct 31 23:23:57.759600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.759568 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ba9088-7469f567cd-f29x9"] Oct 31 23:23:57.941270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.941237 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9cb063-6cf6-4461-87af-5d1e48affa27" path="/var/lib/kubelet/pods/5f9cb063-6cf6-4461-87af-5d1e48affa27/volumes" Oct 31 23:23:57.993741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:57.993714 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:58.265980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:58.265887 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:58.913667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:58.913213 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:59.271363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.271332 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:23:59.305364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305328 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.305606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305530 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.305606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305584 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.305770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305654 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.305770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305701 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.305770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305728 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.305956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.305823 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.306613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.306123 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.306613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.306164 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo\") pod \"be644d97-00b2-4a09-8917-173f8c866aca\" (UID: \"be644d97-00b2-4a09-8917-173f8c866aca\") " Oct 31 23:23:59.309408 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.309356 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.309651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.309627 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.309858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.309838 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.310031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.310003 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.310400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.310372 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:59.311659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.311620 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "kube-api-access-rjgkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.319791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.319755 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.327035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.327011 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:59.327206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.327145 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token" pod "be644d97-00b2-4a09-8917-173f8c866aca" (UID: "be644d97-00b2-4a09-8917-173f8c866aca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.334938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.334911 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:23:59.346627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.344209 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:59.413011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.412980 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413021 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413055 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413105 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413140 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413267 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413313 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413354 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413359 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.413424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413408 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413463 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413574 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413633 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413650 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.413905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413750 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413782 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.413905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413815 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.413905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413849 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.414116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413919 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo\") pod \"9981a666-c637-4f4d-8f48-6e58b9172543\" (UID: \"9981a666-c637-4f4d-8f48-6e58b9172543\") " Oct 31 23:23:59.414116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.413977 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs\") pod \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\" (UID: \"b8a6417a-9af3-4344-86dc-099ba6da3cc8\") " Oct 31 23:23:59.414868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414833 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.414868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414859 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414874 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414890 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be644d97-00b2-4a09-8917-173f8c866aca-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414903 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414917 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414930 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414945 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414959 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be644d97-00b2-4a09-8917-173f8c866aca-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414973 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be644d97-00b2-4a09-8917-173f8c866aca-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.414988 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjgkw\" (UniqueName: \"kubernetes.io/projected/be644d97-00b2-4a09-8917-173f8c866aca-kube-api-access-rjgkw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.415571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.415234 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.415571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.415400 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.415684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.415618 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.415808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.415787 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.415973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.415956 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.416226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.416185 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.418472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.418445 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:59.418716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.418692 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:59.419373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.419219 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.419373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.419334 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.420675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.420613 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:23:59.421066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.421041 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:23:59.421178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.421124 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7" pod "b8a6417a-9af3-4344-86dc-099ba6da3cc8" (UID: "b8a6417a-9af3-4344-86dc-099ba6da3cc8"). InnerVolumeSpecName "kube-api-access-ttnz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.421683 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.421651 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "kube-api-access-wgxj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.422761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.422734 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:23:59.423265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.423239 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token" pod "9981a666-c637-4f4d-8f48-6e58b9172543" (UID: "9981a666-c637-4f4d-8f48-6e58b9172543"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:23:59.516322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516285 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9981a666-c637-4f4d-8f48-6e58b9172543-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516318 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516322 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516329 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516339 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516347 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516357 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516366 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516374 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9981a666-c637-4f4d-8f48-6e58b9172543-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516383 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516392 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgxj8\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-kube-api-access-wgxj8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516402 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttnz7\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-kube-api-access-ttnz7\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516410 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9981a666-c637-4f4d-8f48-6e58b9172543-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516419 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516429 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516437 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9981a666-c637-4f4d-8f48-6e58b9172543-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.516652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.516446 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6417a-9af3-4344-86dc-099ba6da3cc8-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:23:59.622791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.622686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:23:59.727338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.727236 5043 generic.go:356] "Generic (PLEG): container finished" podID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerID="c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c" exitCode=0 Oct 31 23:23:59.727338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.727310 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" Oct 31 23:23:59.727723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.727335 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" event={"ID":"b8a6417a-9af3-4344-86dc-099ba6da3cc8","Type":"ContainerDied","Data":"c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c"} Oct 31 23:23:59.727723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.727504 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln" event={"ID":"b8a6417a-9af3-4344-86dc-099ba6da3cc8","Type":"ContainerDied","Data":"8746e3c177007ffb31de2a37eb51b96d7246d6267b8f022221ef1cd409caa552"} Oct 31 23:23:59.727723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.727567 5043 scope.go:122] "RemoveContainer" containerID="c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c" Oct 31 23:23:59.729421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.729397 5043 generic.go:356] "Generic (PLEG): container finished" podID="be644d97-00b2-4a09-8917-173f8c866aca" containerID="725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15" exitCode=0 Oct 31 23:23:59.729540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.729451 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" Oct 31 23:23:59.729540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.729485 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" event={"ID":"be644d97-00b2-4a09-8917-173f8c866aca","Type":"ContainerDied","Data":"725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15"} Oct 31 23:23:59.729701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.729526 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl" event={"ID":"be644d97-00b2-4a09-8917-173f8c866aca","Type":"ContainerDied","Data":"afb7b1ed281c273b3b0b873470d658294c36ac9301219ec07d6f46760280ade7"} Oct 31 23:23:59.731290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.731267 5043 generic.go:356] "Generic (PLEG): container finished" podID="9981a666-c637-4f4d-8f48-6e58b9172543" containerID="ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645" exitCode=0 Oct 31 23:23:59.731679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.731659 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" event={"ID":"9981a666-c637-4f4d-8f48-6e58b9172543","Type":"ContainerDied","Data":"ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645"} Oct 31 23:23:59.731739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.731708 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" event={"ID":"9981a666-c637-4f4d-8f48-6e58b9172543","Type":"ContainerDied","Data":"7a75973d460039856ea3299e8eccac4a20c9061285f356f45ac5032550200021"} Oct 31 23:23:59.731787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.731774 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm" Oct 31 23:23:59.744490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.744468 5043 scope.go:122] "RemoveContainer" containerID="c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c" Oct 31 23:23:59.744935 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:59.744912 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c\": container with ID starting with c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c not found: ID does not exist" containerID="c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c" Oct 31 23:23:59.745001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.744941 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c"} err="failed to get container status \"c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c\": rpc error: code = NotFound desc = could not find container \"c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c\": container with ID starting with c34baa10a7d7a47af9c423de45a49427a114d2542456a891ecfb49004d1eb67c not found: ID does not exist" Oct 31 23:23:59.745001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.744956 5043 scope.go:122] "RemoveContainer" containerID="725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15" Oct 31 23:23:59.764143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.764108 5043 scope.go:122] "RemoveContainer" containerID="725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15" Oct 31 23:23:59.764504 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:59.764477 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15\": container with ID starting with 725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15 not found: ID does not exist" containerID="725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15" Oct 31 23:23:59.764659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.764510 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15"} err="failed to get container status \"725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15\": rpc error: code = NotFound desc = could not find container \"725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15\": container with ID starting with 725cb19b98a8619b58ea4df1f5b46b487acaedee4a834d07026ef0f7f119eb15 not found: ID does not exist" Oct 31 23:23:59.764659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.764528 5043 scope.go:122] "RemoveContainer" containerID="ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645" Oct 31 23:23:59.769351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.769331 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl"] Oct 31 23:23:59.775344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.775324 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-tfvrl"] Oct 31 23:23:59.776746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.776722 5043 scope.go:122] "RemoveContainer" containerID="ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645" Oct 31 23:23:59.777038 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:23:59.777020 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645\": container with ID starting with ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645 not found: ID does not exist" containerID="ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645" Oct 31 23:23:59.777088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.777047 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645"} err="failed to get container status \"ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645\": rpc error: code = NotFound desc = could not find container \"ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645\": container with ID starting with ba277dff072dcb833cd2fa62f2244fb51c5ff481784833353fb6707e40162645 not found: ID does not exist" Oct 31 23:23:59.782302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.782275 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln"] Oct 31 23:23:59.787157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.787136 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-k6vln"] Oct 31 23:23:59.798671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.798618 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm"] Oct 31 23:23:59.803530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.803508 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-885fdf-openshift-default-758d98b657-z6gmm"] Oct 31 23:23:59.942017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.941977 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" path="/var/lib/kubelet/pods/9981a666-c637-4f4d-8f48-6e58b9172543/volumes" Oct 31 23:23:59.942761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.942739 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" path="/var/lib/kubelet/pods/b8a6417a-9af3-4344-86dc-099ba6da3cc8/volumes" Oct 31 23:23:59.943460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:23:59.943441 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be644d97-00b2-4a09-8917-173f8c866aca" path="/var/lib/kubelet/pods/be644d97-00b2-4a09-8917-173f8c866aca/volumes" Oct 31 23:24:00.391632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:00.391595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:00.457509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:00.457318 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t"] Oct 31 23:24:00.473781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:00.473746 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799"] Oct 31 23:24:00.563517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:00.563480 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:00.732249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:00.732158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:01.272088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:01.272012 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:02.305185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:02.304049 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:03.012899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:03.012861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:03.603504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:03.603319 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:04.283642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.283604 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:04.582928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.582849 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk"] Oct 31 23:24:04.596040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.596006 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb"] Oct 31 23:24:04.692601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.688299 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:04.752726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.752680 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx"] Oct 31 23:24:04.752982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.752952 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" podUID="150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" containerName="maas-api" containerID="cri-o://fef5f76cf68e6867af3954b0c64e9c5e8d39b99ef1daad3a8fd96f4103f84499" gracePeriod=30 Oct 31 23:24:04.826441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.826382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:04.970011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:04.969943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:05.067482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.067445 5043 generic.go:356] "Generic (PLEG): container finished" podID="150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" containerID="fef5f76cf68e6867af3954b0c64e9c5e8d39b99ef1daad3a8fd96f4103f84499" exitCode=0 Oct 31 23:24:05.068224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.067529 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" event={"ID":"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86","Type":"ContainerDied","Data":"fef5f76cf68e6867af3954b0c64e9c5e8d39b99ef1daad3a8fd96f4103f84499"} Oct 31 23:24:05.068224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.067597 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" event={"ID":"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86","Type":"ContainerDied","Data":"0937fe9d38ca23050ad654b7e378de929515517941fe579c6031871fffd938f8"} Oct 31 23:24:05.068224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.067616 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0937fe9d38ca23050ad654b7e378de929515517941fe579c6031871fffd938f8" Oct 31 23:24:05.078735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.078709 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:24:05.134628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.132520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:05.216568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.212182 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\") pod \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " Oct 31 23:24:05.216568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.212282 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs\") pod \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " Oct 31 23:24:05.216568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.212382 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg\" (UniqueName: \"kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg\") pod \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " Oct 31 23:24:05.216568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.213007 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca\") pod \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\" (UID: \"150e336e-e9c8-4c1d-87e3-2f1b61a8ca86\") " Oct 31 23:24:05.253571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.227775 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca" pod "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" (UID: "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:05.317661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.317041 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls" pod "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" (UID: "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:05.334602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.333658 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:05.334602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.333702 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:05.379676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.367911 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:05.548570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.543333 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" podUID="58194d01-fc01-4f35-81ca-b242916ded14" containerName="payload-processing" containerID="cri-o://1b2b71a46463d0e8d384c6c41718dd359fc93eeb14201783726e13eede66e287" gracePeriod=25 Oct 31 23:24:05.548570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.545635 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs" pod "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" (UID: "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:05.573572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.573176 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" podUID="bddb658d-2e37-46b8-a0e4-92d2c28f7039" containerName="payload-pre-processing" containerID="cri-o://5c225d4d5b483d057cfab751c86890ec44759bbc78ed338ead6c4639751d704a" gracePeriod=25 Oct 31 23:24:05.587932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.587349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:05.620568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.617680 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg" pod "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" (UID: "150e336e-e9c8-4c1d-87e3-2f1b61a8ca86"). InnerVolumeSpecName "kube-api-access-x9njg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:05.685893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.661193 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:05.685893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.661225 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9njg\" (UniqueName: \"kubernetes.io/projected/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86-kube-api-access-x9njg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:05.694034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.693496 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs"] Oct 31 23:24:05.694034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.693757 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" containerID="cri-o://803b0c505a846b722a9c109dd0336a41787b7113073da1059a1dcfd4d5f95870" gracePeriod=30 Oct 31 23:24:05.887623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:05.887063 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:06.089855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.089824 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf"] Oct 31 23:24:06.090329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.090086 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" containerID="cri-o://e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9" gracePeriod=30 Oct 31 23:24:06.092495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.092214 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:06.107997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.107964 5043 generic.go:356] "Generic (PLEG): container finished" podID="58194d01-fc01-4f35-81ca-b242916ded14" containerID="1b2b71a46463d0e8d384c6c41718dd359fc93eeb14201783726e13eede66e287" exitCode=0 Oct 31 23:24:06.108152 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.108104 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" event={"ID":"58194d01-fc01-4f35-81ca-b242916ded14","Type":"ContainerDied","Data":"1b2b71a46463d0e8d384c6c41718dd359fc93eeb14201783726e13eede66e287"} Oct 31 23:24:06.110820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.110791 5043 generic.go:356] "Generic (PLEG): container finished" podID="bddb658d-2e37-46b8-a0e4-92d2c28f7039" containerID="5c225d4d5b483d057cfab751c86890ec44759bbc78ed338ead6c4639751d704a" exitCode=0 Oct 31 23:24:06.111827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.111732 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx" Oct 31 23:24:06.113393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.113306 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" event={"ID":"bddb658d-2e37-46b8-a0e4-92d2c28f7039","Type":"ContainerDied","Data":"5c225d4d5b483d057cfab751c86890ec44759bbc78ed338ead6c4639751d704a"} Oct 31 23:24:06.311015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.310109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:06.312693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.312661 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx"] Oct 31 23:24:06.378869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.376146 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a1deb6-5bbc64968d-mw7vx"] Oct 31 23:24:06.383360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.383314 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw"] Oct 31 23:24:06.384501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384474 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" Oct 31 23:24:06.384501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384500 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384524 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384532 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384604 5043 cpu_manager.go:414] "containerMap: removing container" podUID="150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" containerName="maas-api" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384612 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" containerName="maas-api" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384645 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384652 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384663 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384670 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384681 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5f9cb063-6cf6-4461-87af-5d1e48affa27" containerName="maas-api" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384688 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9cb063-6cf6-4461-87af-5d1e48affa27" containerName="maas-api" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384699 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5ad03db6-9f68-4499-abea-aa85bc145b09" containerName="revoke-keys" Oct 31 23:24:06.384775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384706 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad03db6-9f68-4499-abea-aa85bc145b09" containerName="revoke-keys" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384892 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9981a666-c637-4f4d-8f48-6e58b9172543" containerName="istio-proxy" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384904 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be644d97-00b2-4a09-8917-173f8c866aca" containerName="istio-proxy" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384923 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" containerName="maas-api" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384936 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a231fc23-2c9b-4337-bc59-fd95964c2a2f" containerName="istio-proxy" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384945 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a6417a-9af3-4344-86dc-099ba6da3cc8" containerName="istio-proxy" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384956 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9cb063-6cf6-4461-87af-5d1e48affa27" containerName="maas-api" Oct 31 23:24:06.385390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.384965 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad03db6-9f68-4499-abea-aa85bc145b09" containerName="revoke-keys" Oct 31 23:24:06.390959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.390937 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.395587 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.395533 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw"] Oct 31 23:24:06.395715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.395690 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:06.395779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.395723 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:06.489602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.487623 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.489602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.487718 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.489602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.487794 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8ksm\" (UniqueName: \"kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.505846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.505806 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p"] Oct 31 23:24:06.619639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.602467 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.619639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.603476 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.619639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.603607 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8ksm\" (UniqueName: \"kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.619639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.603360 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.619639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.611321 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:06.638818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.636013 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.717607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.711707 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8ksm\" (UniqueName: \"kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.731656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.726246 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:06.813574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.793518 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p"] Oct 31 23:24:06.941936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.918690 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:06.941936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.918858 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:06.941936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.918986 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5nw\" (UniqueName: \"kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:06.986166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:06.986099 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:24:07.053115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.053019 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume\") pod \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " Oct 31 23:24:07.053792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.053364 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.053792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.053491 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.053792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.053584 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5nw\" (UniqueName: \"kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.056857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.056825 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume" pod "bddb658d-2e37-46b8-a0e4-92d2c28f7039" (UID: "bddb658d-2e37-46b8-a0e4-92d2c28f7039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:07.058192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.058163 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.058966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.058936 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.086347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.086218 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:24:07.099354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.098350 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5nw\" (UniqueName: \"kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.182786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.181629 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume\") pod \"58194d01-fc01-4f35-81ca-b242916ded14\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " Oct 31 23:24:07.182786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.181792 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth\" (UniqueName: \"kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth\") pod \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\" (UID: \"bddb658d-2e37-46b8-a0e4-92d2c28f7039\") " Oct 31 23:24:07.182786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.181890 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2\" (UniqueName: \"kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2\") pod \"58194d01-fc01-4f35-81ca-b242916ded14\" (UID: \"58194d01-fc01-4f35-81ca-b242916ded14\") " Oct 31 23:24:07.182786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.182421 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume" pod "58194d01-fc01-4f35-81ca-b242916ded14" (UID: "58194d01-fc01-4f35-81ca-b242916ded14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:07.182786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.182570 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58194d01-fc01-4f35-81ca-b242916ded14-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:07.182786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.182592 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bddb658d-2e37-46b8-a0e4-92d2c28f7039-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:07.196786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.195704 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2" pod "58194d01-fc01-4f35-81ca-b242916ded14" (UID: "58194d01-fc01-4f35-81ca-b242916ded14"). InnerVolumeSpecName "kube-api-access-hq9b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:07.196786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.195764 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth" pod "bddb658d-2e37-46b8-a0e4-92d2c28f7039" (UID: "bddb658d-2e37-46b8-a0e4-92d2c28f7039"). InnerVolumeSpecName "kube-api-access-2tjth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:07.238183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.238139 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" event={"ID":"58194d01-fc01-4f35-81ca-b242916ded14","Type":"ContainerDied","Data":"59c8fb40201f2905179c3ac9b7517482c1dd48443a705ed4234106a7012fffea"} Oct 31 23:24:07.238492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.238474 5043 scope.go:122] "RemoveContainer" containerID="1b2b71a46463d0e8d384c6c41718dd359fc93eeb14201783726e13eede66e287" Oct 31 23:24:07.238862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.238840 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t" Oct 31 23:24:07.279583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.260848 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:07.279583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.279476 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw"] Oct 31 23:24:07.282163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.282107 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" event={"ID":"bddb658d-2e37-46b8-a0e4-92d2c28f7039","Type":"ContainerDied","Data":"874d432523443ad5821aa9bea61a7f9d281d3fb457df15b10dc8b73004e2e0a2"} Oct 31 23:24:07.282273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.282234 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799" Oct 31 23:24:07.284947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.284861 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:07.284947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.284902 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjth\" (UniqueName: \"kubernetes.io/projected/bddb658d-2e37-46b8-a0e4-92d2c28f7039-kube-api-access-2tjth\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:07.284947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.284921 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq9b2\" (UniqueName: \"kubernetes.io/projected/58194d01-fc01-4f35-81ca-b242916ded14-kube-api-access-hq9b2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:07.290234 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:07.289254 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcda8e48_b494_4fd1_b435_4780fe793cce.slice/crio-bd23e9e426dc160cf1045fba3d4e4087083b0aadcbf7b039d79530d91dc9f45c WatchSource:0}: Error finding container bd23e9e426dc160cf1045fba3d4e4087083b0aadcbf7b039d79530d91dc9f45c: Status 404 returned error can't find the container with id bd23e9e426dc160cf1045fba3d4e4087083b0aadcbf7b039d79530d91dc9f45c Oct 31 23:24:07.298053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.296928 5043 scope.go:122] "RemoveContainer" containerID="5c225d4d5b483d057cfab751c86890ec44759bbc78ed338ead6c4639751d704a" Oct 31 23:24:07.375928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.375692 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t"] Oct 31 23:24:07.380845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.380824 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-836c99-55cb65fd4d-8sm7t"] Oct 31 23:24:07.388313 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.388265 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799"] Oct 31 23:24:07.395486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.395462 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-836c99-76995b99f-f4799"] Oct 31 23:24:07.710328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.710299 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p"] Oct 31 23:24:07.710444 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:07.710365 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641ada32_ef23_4611_99fa_183878a6d579.slice/crio-ab615fbe8383dd80d5f629927de3cc60e544992fd5c5b90ed3f3c534a14223d6 WatchSource:0}: Error finding container ab615fbe8383dd80d5f629927de3cc60e544992fd5c5b90ed3f3c534a14223d6: Status 404 returned error can't find the container with id ab615fbe8383dd80d5f629927de3cc60e544992fd5c5b90ed3f3c534a14223d6 Oct 31 23:24:07.895520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.895480 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:07.990302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.990270 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150e336e-e9c8-4c1d-87e3-2f1b61a8ca86" path="/var/lib/kubelet/pods/150e336e-e9c8-4c1d-87e3-2f1b61a8ca86/volumes" Oct 31 23:24:07.991002 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.990977 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58194d01-fc01-4f35-81ca-b242916ded14" path="/var/lib/kubelet/pods/58194d01-fc01-4f35-81ca-b242916ded14/volumes" Oct 31 23:24:07.991488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:07.991466 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddb658d-2e37-46b8-a0e4-92d2c28f7039" path="/var/lib/kubelet/pods/bddb658d-2e37-46b8-a0e4-92d2c28f7039/volumes" Oct 31 23:24:08.295199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.295081 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" event={"ID":"641ada32-ef23-4611-99fa-183878a6d579","Type":"ContainerStarted","Data":"537cdf640cd9d329c940d4e516882aecb9a112aca25a1a18d2dbd11de259ccce"} Oct 31 23:24:08.295199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.295148 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" event={"ID":"641ada32-ef23-4611-99fa-183878a6d579","Type":"ContainerStarted","Data":"ab615fbe8383dd80d5f629927de3cc60e544992fd5c5b90ed3f3c534a14223d6"} Oct 31 23:24:08.300285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.300234 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" event={"ID":"fcda8e48-b494-4fd1-b435-4780fe793cce","Type":"ContainerStarted","Data":"61584c5fc7c47a20f74390d78f444e55032a49cbb0b48398c6a8988194d7f52e"} Oct 31 23:24:08.300451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.300288 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" event={"ID":"fcda8e48-b494-4fd1-b435-4780fe793cce","Type":"ContainerStarted","Data":"bd23e9e426dc160cf1045fba3d4e4087083b0aadcbf7b039d79530d91dc9f45c"} Oct 31 23:24:08.359161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.359106 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" podStartSLOduration=2.359091319 podStartE2EDuration="2.359091319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:08.313453143 +0000 UTC m=+2074.543007555" watchObservedRunningTime="2026-10-31 23:24:08.359091319 +0000 UTC m=+2074.588645729" Oct 31 23:24:08.364295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.364247 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" podStartSLOduration=2.364233737 podStartE2EDuration="2.364233737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:08.360525515 +0000 UTC m=+2074.590079927" watchObservedRunningTime="2026-10-31 23:24:08.364233737 +0000 UTC m=+2074.593788147" Oct 31 23:24:08.666628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:08.651204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:09.312178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.312143 5043 generic.go:356] "Generic (PLEG): container finished" podID="641ada32-ef23-4611-99fa-183878a6d579" containerID="537cdf640cd9d329c940d4e516882aecb9a112aca25a1a18d2dbd11de259ccce" exitCode=0 Oct 31 23:24:09.312639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.312219 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" event={"ID":"641ada32-ef23-4611-99fa-183878a6d579","Type":"ContainerDied","Data":"537cdf640cd9d329c940d4e516882aecb9a112aca25a1a18d2dbd11de259ccce"} Oct 31 23:24:09.329568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.322807 5043 generic.go:356] "Generic (PLEG): container finished" podID="fcda8e48-b494-4fd1-b435-4780fe793cce" containerID="61584c5fc7c47a20f74390d78f444e55032a49cbb0b48398c6a8988194d7f52e" exitCode=0 Oct 31 23:24:09.329568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.322873 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" event={"ID":"fcda8e48-b494-4fd1-b435-4780fe793cce","Type":"ContainerDied","Data":"61584c5fc7c47a20f74390d78f444e55032a49cbb0b48398c6a8988194d7f52e"} Oct 31 23:24:09.407878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.407809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:09.584211 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.584120 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" containerID="cri-o://ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f" gracePeriod=25 Oct 31 23:24:09.596686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.596648 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" containerID="cri-o://90e64e438bd45a3b37d8dc8a0f2035c3c8c2b25943e3e7281e357f39ed4e628c" gracePeriod=25 Oct 31 23:24:09.925909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:09.925762 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:10.141129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.137174 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:10.322570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.321524 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:10.376435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.376395 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.379095 5043 generic.go:356] "Generic (PLEG): container finished" podID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerID="ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f" exitCode=0 Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.379397 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" event={"ID":"7e1072fa-4c18-4b72-a947-325fa267a72f","Type":"ContainerDied","Data":"ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f"} Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.379493 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" event={"ID":"7e1072fa-4c18-4b72-a947-325fa267a72f","Type":"ContainerDied","Data":"c3060fe7f2f40501d646e80f948f10e277932b6f23ce8f5de857c33c1da6fe8b"} Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.379534 5043 scope.go:122] "RemoveContainer" containerID="ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f" Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.384699 5043 generic.go:356] "Generic (PLEG): container finished" podID="3db4e830-7446-4841-9080-f4572baaecca" containerID="90e64e438bd45a3b37d8dc8a0f2035c3c8c2b25943e3e7281e357f39ed4e628c" exitCode=0 Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.385346 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" event={"ID":"3db4e830-7446-4841-9080-f4572baaecca","Type":"ContainerDied","Data":"90e64e438bd45a3b37d8dc8a0f2035c3c8c2b25943e3e7281e357f39ed4e628c"} Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.385393 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" event={"ID":"3db4e830-7446-4841-9080-f4572baaecca","Type":"ContainerDied","Data":"5507131c7ff3f28eb790d772235502741f6da87f7eee3f73f2c790cfda96a6f9"} Oct 31 23:24:10.401957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.385445 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5507131c7ff3f28eb790d772235502741f6da87f7eee3f73f2c790cfda96a6f9" Oct 31 23:24:10.505723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.491119 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:24:10.505723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.503160 5043 scope.go:122] "RemoveContainer" containerID="ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f" Oct 31 23:24:10.505723 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:10.503627 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f\": container with ID starting with ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f not found: ID does not exist" containerID="ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f" Oct 31 23:24:10.505723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.503676 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f"} err="failed to get container status \"ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f\": rpc error: code = NotFound desc = could not find container \"ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f\": container with ID starting with ffc4b2a242d7773766b8b694811d8b4c1fee9c5fafd8aa705a09a83c95aea50f not found: ID does not exist" Oct 31 23:24:10.574569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.567600 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:10.584576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.579400 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume\") pod \"7e1072fa-4c18-4b72-a947-325fa267a72f\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " Oct 31 23:24:10.584576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.580048 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume" pod "7e1072fa-4c18-4b72-a947-325fa267a72f" (UID: "7e1072fa-4c18-4b72-a947-325fa267a72f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:10.584576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.580671 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq\" (UniqueName: \"kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq\") pod \"7e1072fa-4c18-4b72-a947-325fa267a72f\" (UID: \"7e1072fa-4c18-4b72-a947-325fa267a72f\") " Oct 31 23:24:10.584875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.584621 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1072fa-4c18-4b72-a947-325fa267a72f-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.598462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.594202 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq" pod "7e1072fa-4c18-4b72-a947-325fa267a72f" (UID: "7e1072fa-4c18-4b72-a947-325fa267a72f"). InnerVolumeSpecName "kube-api-access-dt5bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:10.598462 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:10.597597 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:24:10.694615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.688973 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j\" (UniqueName: \"kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j\") pod \"3db4e830-7446-4841-9080-f4572baaecca\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " Oct 31 23:24:10.694615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.689024 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume\") pod \"3db4e830-7446-4841-9080-f4572baaecca\" (UID: \"3db4e830-7446-4841-9080-f4572baaecca\") " Oct 31 23:24:10.694615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.689828 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5bq\" (UniqueName: \"kubernetes.io/projected/7e1072fa-4c18-4b72-a947-325fa267a72f-kube-api-access-dt5bq\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.694615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.690659 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume" pod "3db4e830-7446-4841-9080-f4572baaecca" (UID: "3db4e830-7446-4841-9080-f4572baaecca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:10.694615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.693127 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j" pod "3db4e830-7446-4841-9080-f4572baaecca" (UID: "3db4e830-7446-4841-9080-f4572baaecca"). InnerVolumeSpecName "kube-api-access-6jz6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:10.798409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.798293 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jz6j\" (UniqueName: \"kubernetes.io/projected/3db4e830-7446-4841-9080-f4572baaecca-kube-api-access-6jz6j\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.798409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.798333 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db4e830-7446-4841-9080-f4572baaecca-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:10.869236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.869198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:10.961858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.961787 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:10.975429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:10.975407 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:11.001786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.001754 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca\") pod \"641ada32-ef23-4611-99fa-183878a6d579\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " Oct 31 23:24:11.001928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.001823 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca\") pod \"fcda8e48-b494-4fd1-b435-4780fe793cce\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " Oct 31 23:24:11.001928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.001901 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca\") pod \"fcda8e48-b494-4fd1-b435-4780fe793cce\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " Oct 31 23:24:11.002063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.001990 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca\") pod \"641ada32-ef23-4611-99fa-183878a6d579\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " Oct 31 23:24:11.002063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.002051 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm\" (UniqueName: \"kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm\") pod \"fcda8e48-b494-4fd1-b435-4780fe793cce\" (UID: \"fcda8e48-b494-4fd1-b435-4780fe793cce\") " Oct 31 23:24:11.003067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.002272 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw\" (UniqueName: \"kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw\") pod \"641ada32-ef23-4611-99fa-183878a6d579\" (UID: \"641ada32-ef23-4611-99fa-183878a6d579\") " Oct 31 23:24:11.003067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.003001 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca" pod "641ada32-ef23-4611-99fa-183878a6d579" (UID: "641ada32-ef23-4611-99fa-183878a6d579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:11.006707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.003484 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641ada32-ef23-4611-99fa-183878a6d579-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.006707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.006201 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca" pod "fcda8e48-b494-4fd1-b435-4780fe793cce" (UID: "fcda8e48-b494-4fd1-b435-4780fe793cce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:11.006707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.006490 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw" pod "641ada32-ef23-4611-99fa-183878a6d579" (UID: "641ada32-ef23-4611-99fa-183878a6d579"). InnerVolumeSpecName "kube-api-access-4h5nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.007109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.007088 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca" pod "fcda8e48-b494-4fd1-b435-4780fe793cce" (UID: "fcda8e48-b494-4fd1-b435-4780fe793cce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:11.008529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.008508 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm" pod "fcda8e48-b494-4fd1-b435-4780fe793cce" (UID: "fcda8e48-b494-4fd1-b435-4780fe793cce"). InnerVolumeSpecName "kube-api-access-x8ksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.011387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.011367 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca" pod "641ada32-ef23-4611-99fa-183878a6d579" (UID: "641ada32-ef23-4611-99fa-183878a6d579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:11.104692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.104660 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5nw\" (UniqueName: \"kubernetes.io/projected/641ada32-ef23-4611-99fa-183878a6d579-kube-api-access-4h5nw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.104692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.104691 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641ada32-ef23-4611-99fa-183878a6d579-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.104894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.104705 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.104894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.104718 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcda8e48-b494-4fd1-b435-4780fe793cce-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.104894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.104732 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8ksm\" (UniqueName: \"kubernetes.io/projected/fcda8e48-b494-4fd1-b435-4780fe793cce-kube-api-access-x8ksm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.198756 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:11.198721 5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc769ec16_9843_4e45_9caf_298744983e65.slice/crio-conmon-e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc769ec16_9843_4e45_9caf_298744983e65.slice/crio-e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9.scope\": RecentStats: unable to find data in memory cache]" Oct 31 23:24:11.310129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.307691 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:11.444094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.443851 5043 generic.go:356] "Generic (PLEG): container finished" podID="c769ec16-9843-4e45-9caf-298744983e65" containerID="e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9" exitCode=0 Oct 31 23:24:11.444094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.444047 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" event={"ID":"c769ec16-9843-4e45-9caf-298744983e65","Type":"ContainerDied","Data":"e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9"} Oct 31 23:24:11.444504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.444123 5043 scope.go:122] "RemoveContainer" containerID="78852d1ffea1e452bca13c755e516b7def266dd3b199c930c73723528e76076c" Oct 31 23:24:11.448447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.447965 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk" Oct 31 23:24:11.450805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.450676 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" event={"ID":"fcda8e48-b494-4fd1-b435-4780fe793cce","Type":"ContainerDied","Data":"bd23e9e426dc160cf1045fba3d4e4087083b0aadcbf7b039d79530d91dc9f45c"} Oct 31 23:24:11.450805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.450729 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd23e9e426dc160cf1045fba3d4e4087083b0aadcbf7b039d79530d91dc9f45c" Oct 31 23:24:11.450805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.450801 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw" Oct 31 23:24:11.456990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.456963 5043 generic.go:356] "Generic (PLEG): container finished" podID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerID="803b0c505a846b722a9c109dd0336a41787b7113073da1059a1dcfd4d5f95870" exitCode=0 Oct 31 23:24:11.457190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.457174 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" event={"ID":"4ce8af1c-a77d-44b2-920d-9d57b20a21ad","Type":"ContainerDied","Data":"803b0c505a846b722a9c109dd0336a41787b7113073da1059a1dcfd4d5f95870"} Oct 31 23:24:11.459018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.458999 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb" Oct 31 23:24:11.459785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.459249 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" Oct 31 23:24:11.465379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.464668 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p" event={"ID":"641ada32-ef23-4611-99fa-183878a6d579","Type":"ContainerDied","Data":"ab615fbe8383dd80d5f629927de3cc60e544992fd5c5b90ed3f3c534a14223d6"} Oct 31 23:24:11.465379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.464720 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab615fbe8383dd80d5f629927de3cc60e544992fd5c5b90ed3f3c534a14223d6" Oct 31 23:24:11.494902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.494835 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk"] Oct 31 23:24:11.505490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.505462 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2126ff-58b77bf5dd-zjppk"] Oct 31 23:24:11.522104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.522080 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb"] Oct 31 23:24:11.530926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.529833 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2126ff-55f7fffdc5-z92tb"] Oct 31 23:24:11.557926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.557883 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:24:11.621695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.621658 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.621946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.621929 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.622069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.622054 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.622197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.622185 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.622437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.622420 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.622635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.622621 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.622833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.622818 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.623030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.623017 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.623170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.623157 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj\") pod \"c769ec16-9843-4e45-9caf-298744983e65\" (UID: \"c769ec16-9843-4e45-9caf-298744983e65\") " Oct 31 23:24:11.629469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.629433 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.629908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.629864 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.630016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.629902 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.630193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.630171 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.630278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.630208 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:11.630581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.630558 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.631254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.631227 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "kube-api-access-v79mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.632395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.632371 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:11.633650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.633592 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token" pod "c769ec16-9843-4e45-9caf-298744983e65" (UID: "c769ec16-9843-4e45-9caf-298744983e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.643574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.643532 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:24:11.725352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.725315 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.726058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.726032 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.726631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.726457 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.726778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.726755 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.726864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.726848 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.726938 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.726923 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.727042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.726994 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.727100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727067 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.727153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727110 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.727207 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727190 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.727268 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727240 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo\") pod \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\" (UID: \"4ce8af1c-a77d-44b2-920d-9d57b20a21ad\") " Oct 31 23:24:11.727455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727430 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.727956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727931 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727960 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c769ec16-9843-4e45-9caf-298744983e65-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727977 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79mj\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-kube-api-access-v79mj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.727994 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c769ec16-9843-4e45-9caf-298744983e65-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728008 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728022 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728035 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728050 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728064 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728078 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c769ec16-9843-4e45-9caf-298744983e65-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728093 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c769ec16-9843-4e45-9caf-298744983e65-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728107 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.728407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728164 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.728407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728400 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.728759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.728525 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:11.730356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.730322 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:11.730491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.730472 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "kube-api-access-22xfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:11.731820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.731798 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo" pod "4ce8af1c-a77d-44b2-920d-9d57b20a21ad" (UID: "4ce8af1c-a77d-44b2-920d-9d57b20a21ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:11.830014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.829941 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.830014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.829977 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22xfx\" (UniqueName: \"kubernetes.io/projected/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-kube-api-access-22xfx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.830014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.829993 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.830014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.830007 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.830240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.830021 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.830240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.830035 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ce8af1c-a77d-44b2-920d-9d57b20a21ad-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:11.939333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.939296 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db4e830-7446-4841-9080-f4572baaecca" path="/var/lib/kubelet/pods/3db4e830-7446-4841-9080-f4572baaecca/volumes" Oct 31 23:24:11.939924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:11.939905 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" path="/var/lib/kubelet/pods/7e1072fa-4c18-4b72-a947-325fa267a72f/volumes" Oct 31 23:24:12.082971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.082882 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:12.471796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.471725 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" Oct 31 23:24:12.471796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.471772 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs" event={"ID":"4ce8af1c-a77d-44b2-920d-9d57b20a21ad","Type":"ContainerDied","Data":"85ac2c632c6e385ead346ff34dd220198cd20b0b1bddcfa813a1079662409ffa"} Oct 31 23:24:12.472234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.471844 5043 scope.go:122] "RemoveContainer" containerID="803b0c505a846b722a9c109dd0336a41787b7113073da1059a1dcfd4d5f95870" Oct 31 23:24:12.473616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.473591 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" event={"ID":"c769ec16-9843-4e45-9caf-298744983e65","Type":"ContainerDied","Data":"9189f1dc092962f11a0efe3ab9c6571ea2abd46dba93685f978cd4dd39432ec2"} Oct 31 23:24:12.473747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.473632 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf" Oct 31 23:24:12.484862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.484843 5043 scope.go:122] "RemoveContainer" containerID="e3d6b38130d0230ad6dc5dba5f53bb98cfea3f45d35f9b0329c3d53ecbddb1e9" Oct 31 23:24:12.501591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.501562 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf"] Oct 31 23:24:12.516530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.516504 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ba9088-openshift-default-6869694875-6klsf"] Oct 31 23:24:12.529366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.529335 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs"] Oct 31 23:24:12.537441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.537416 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8e6a08-openshift-default-cfb9d6955-v2mrs"] Oct 31 23:24:12.706606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.703528 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:12.966566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:12.966057 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:13.123579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.121979 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:13.245620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.232834 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv"] Oct 31 23:24:13.245620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.233086 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" containerID="cri-o://48dcced70c7a43a399e0827691f5be6beb8a7434633296cbc8c2964b64eacce5" gracePeriod=30 Oct 31 23:24:13.245620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.240650 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt"] Oct 31 23:24:13.245620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.240909 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" containerID="cri-o://b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04" gracePeriod=30 Oct 31 23:24:13.304916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.304877 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:13.466568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.458970 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:13.833989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.833951 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc"] Oct 31 23:24:13.835604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835573 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" Oct 31 23:24:13.835604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835600 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835638 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835646 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835680 5043 cpu_manager.go:414] "containerMap: removing container" podUID="58194d01-fc01-4f35-81ca-b242916ded14" containerName="payload-processing" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835690 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58194d01-fc01-4f35-81ca-b242916ded14" containerName="payload-processing" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835708 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fcda8e48-b494-4fd1-b435-4780fe793cce" containerName="revoke-keys" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835717 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcda8e48-b494-4fd1-b435-4780fe793cce" containerName="revoke-keys" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835751 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835758 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835791 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835798 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835810 5043 cpu_manager.go:414] "containerMap: removing container" podUID="641ada32-ef23-4611-99fa-183878a6d579" containerName="revoke-keys" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835817 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641ada32-ef23-4611-99fa-183878a6d579" containerName="revoke-keys" Oct 31 23:24:13.835823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835829 5043 cpu_manager.go:414] "containerMap: removing container" podUID="bddb658d-2e37-46b8-a0e4-92d2c28f7039" containerName="payload-pre-processing" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.835836 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddb658d-2e37-46b8-a0e4-92d2c28f7039" containerName="payload-pre-processing" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836022 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1072fa-4c18-4b72-a947-325fa267a72f" containerName="payload-processing" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836037 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="641ada32-ef23-4611-99fa-183878a6d579" containerName="revoke-keys" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836047 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddb658d-2e37-46b8-a0e4-92d2c28f7039" containerName="payload-pre-processing" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836057 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836073 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" containerName="istio-proxy" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836088 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="58194d01-fc01-4f35-81ca-b242916ded14" containerName="payload-processing" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836100 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcda8e48-b494-4fd1-b435-4780fe793cce" containerName="revoke-keys" Oct 31 23:24:13.836518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.836116 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db4e830-7446-4841-9080-f4572baaecca" containerName="payload-pre-processing" Oct 31 23:24:13.846121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.846093 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:13.849246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.847945 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc"] Oct 31 23:24:13.851736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.851696 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:13.851899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.851882 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:13.941819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.941767 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce8af1c-a77d-44b2-920d-9d57b20a21ad" path="/var/lib/kubelet/pods/4ce8af1c-a77d-44b2-920d-9d57b20a21ad/volumes" Oct 31 23:24:13.942447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.942430 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c769ec16-9843-4e45-9caf-298744983e65" path="/var/lib/kubelet/pods/c769ec16-9843-4e45-9caf-298744983e65/volumes" Oct 31 23:24:13.947500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.947031 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz28k\" (UniqueName: \"kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:13.947500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.947132 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:13.947500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:13.947280 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.049077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.049033 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.049355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.049245 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz28k\" (UniqueName: \"kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.049455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.049359 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.049913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.049884 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.052463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.052435 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.064495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.064444 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz28k\" (UniqueName: \"kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k\") pod \"maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.135196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.135084 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:14.135196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.135144 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:14.192897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.192865 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:14.402641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.402472 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:14.549589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.548380 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:14.565603 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:14.564496 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39da29dc_5612_402f_b061_727435a6974c.slice/crio-2326ed7b9771f1266cb3c51979260d84163ea13f0fc6cf141c904297f8f04ccd WatchSource:0}: Error finding container 2326ed7b9771f1266cb3c51979260d84163ea13f0fc6cf141c904297f8f04ccd: Status 404 returned error can't find the container with id 2326ed7b9771f1266cb3c51979260d84163ea13f0fc6cf141c904297f8f04ccd Oct 31 23:24:14.565748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:14.565701 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc"] Oct 31 23:24:15.112285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.112227 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:15.310078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.310040 5043 patch_prober.go:29] interesting pod/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:15.310236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.310108 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:15.551790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.551751 5043 generic.go:356] "Generic (PLEG): container finished" podID="39da29dc-5612-402f-b061-727435a6974c" containerID="a61b815473f03bd142ad2b28ae9df7c9423c7928a316f16d5267449a160524a3" exitCode=0 Oct 31 23:24:15.552100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.552082 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" event={"ID":"39da29dc-5612-402f-b061-727435a6974c","Type":"ContainerDied","Data":"a61b815473f03bd142ad2b28ae9df7c9423c7928a316f16d5267449a160524a3"} Oct 31 23:24:15.552181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.552167 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" event={"ID":"39da29dc-5612-402f-b061-727435a6974c","Type":"ContainerStarted","Data":"2326ed7b9771f1266cb3c51979260d84163ea13f0fc6cf141c904297f8f04ccd"} Oct 31 23:24:15.962087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:15.962054 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:16.435276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:16.435189 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:16.572455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:16.570809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:16.815568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:16.812981 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4"] Oct 31 23:24:16.852696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:16.838052 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:17.064569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.064070 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:17.317592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.314612 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:17.405871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.405834 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k\" (UniqueName: \"kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k\") pod \"39da29dc-5612-402f-b061-727435a6974c\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " Oct 31 23:24:17.406087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.405988 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca\") pod \"39da29dc-5612-402f-b061-727435a6974c\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " Oct 31 23:24:17.406087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.406069 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca\") pod \"39da29dc-5612-402f-b061-727435a6974c\" (UID: \"39da29dc-5612-402f-b061-727435a6974c\") " Oct 31 23:24:17.406399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.406376 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca" pod "39da29dc-5612-402f-b061-727435a6974c" (UID: "39da29dc-5612-402f-b061-727435a6974c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:17.407637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.406855 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39da29dc-5612-402f-b061-727435a6974c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:17.408581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.408527 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k" pod "39da29dc-5612-402f-b061-727435a6974c" (UID: "39da29dc-5612-402f-b061-727435a6974c"). InnerVolumeSpecName "kube-api-access-jz28k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:17.408788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.408764 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca" pod "39da29dc-5612-402f-b061-727435a6974c" (UID: "39da29dc-5612-402f-b061-727435a6974c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:17.480086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.480058 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:17.508241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.508212 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39da29dc-5612-402f-b061-727435a6974c-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:17.508241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.508241 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz28k\" (UniqueName: \"kubernetes.io/projected/39da29dc-5612-402f-b061-727435a6974c-kube-api-access-jz28k\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:17.579240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.579208 5043 generic.go:356] "Generic (PLEG): container finished" podID="74083906-8df2-4339-9148-e0d1f5c4de69" containerID="4211256ebe40147676ce3dc6bd532c51a736f550e2e6f98e987f48f8c8b361e7" exitCode=0 Oct 31 23:24:17.579441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.579282 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" event={"ID":"74083906-8df2-4339-9148-e0d1f5c4de69","Type":"ContainerDied","Data":"4211256ebe40147676ce3dc6bd532c51a736f550e2e6f98e987f48f8c8b361e7"} Oct 31 23:24:17.579441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.579340 5043 scope.go:122] "RemoveContainer" containerID="43d680ee9d1ab3f662447e423976d1a941a1f04d2a364349d62ca1799ea498a5" Oct 31 23:24:17.579875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.579854 5043 scope.go:122] "RemoveContainer" containerID="4211256ebe40147676ce3dc6bd532c51a736f550e2e6f98e987f48f8c8b361e7" Oct 31 23:24:17.580687 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:17.580291 5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr_openshift-ingress(74083906-8df2-4339-9148-e0d1f5c4de69)\"" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" Oct 31 23:24:17.581083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.581057 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" event={"ID":"39da29dc-5612-402f-b061-727435a6974c","Type":"ContainerDied","Data":"2326ed7b9771f1266cb3c51979260d84163ea13f0fc6cf141c904297f8f04ccd"} Oct 31 23:24:17.581192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.581097 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2326ed7b9771f1266cb3c51979260d84163ea13f0fc6cf141c904297f8f04ccd" Oct 31 23:24:17.581192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.581070 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc" Oct 31 23:24:17.603162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.603136 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p"] Oct 31 23:24:17.609127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.609107 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f80dfa-bmp2p"] Oct 31 23:24:17.646568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.645724 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:17.796208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.796092 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:17.946571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.946461 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:17.996570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:17.996232 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641ada32-ef23-4611-99fa-183878a6d579" path="/var/lib/kubelet/pods/641ada32-ef23-4611-99fa-183878a6d579/volumes" Oct 31 23:24:18.243622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.235318 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:18.276578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.272582 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr"] Oct 31 23:24:18.412813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.407148 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:18.629444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.629406 5043 generic.go:356] "Generic (PLEG): container finished" podID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerID="48dcced70c7a43a399e0827691f5be6beb8a7434633296cbc8c2964b64eacce5" exitCode=0 Oct 31 23:24:18.630102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.630078 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" event={"ID":"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0","Type":"ContainerDied","Data":"48dcced70c7a43a399e0827691f5be6beb8a7434633296cbc8c2964b64eacce5"} Oct 31 23:24:18.630312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.630297 5043 scope.go:122] "RemoveContainer" containerID="6b83c0f880f764c89884bd5c57afb6261257f1ebdc75e46acd9c162938987327" Oct 31 23:24:18.691010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.690972 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst"] Oct 31 23:24:18.693158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.692293 5043 cpu_manager.go:414] "containerMap: removing container" podUID="39da29dc-5612-402f-b061-727435a6974c" containerName="revoke-keys" Oct 31 23:24:18.693158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.692312 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39da29dc-5612-402f-b061-727435a6974c" containerName="revoke-keys" Oct 31 23:24:18.693158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.692341 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" Oct 31 23:24:18.693158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.692350 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" Oct 31 23:24:18.693158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.692528 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="39da29dc-5612-402f-b061-727435a6974c" containerName="revoke-keys" Oct 31 23:24:18.693158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.692558 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c769ec16-9843-4e45-9caf-298744983e65" containerName="istio-proxy" Oct 31 23:24:18.701647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.700811 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.705570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.703943 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst"] Oct 31 23:24:18.705692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.705679 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:18.705760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.705751 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:18.812481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.811005 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.812481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.811250 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.812481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.811456 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.933195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.913612 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.933195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.913714 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.933195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.913915 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.933195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.914853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.964668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.948987 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:18.964668 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:18.962047 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:19.005243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.005220 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:24:19.008884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.008860 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:24:19.079115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.078154 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:19.090590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.084749 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:19.179150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.179046 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.210826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.210795 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.211122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211108 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.211210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211201 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.211300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211291 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.211372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211364 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.211502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211494 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.211637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211620 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.211798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211780 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.211990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.211954 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.212118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212101 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.212242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212226 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.212362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212350 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.212448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212439 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert\") pod \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\" (UID: \"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0\") " Oct 31 23:24:19.212538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212527 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.212685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212672 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.212834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212824 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.212930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.212921 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket\") pod \"74083906-8df2-4339-9148-e0d1f5c4de69\" (UID: \"74083906-8df2-4339-9148-e0d1f5c4de69\") " Oct 31 23:24:19.232708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.185036 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:19.253660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.253630 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.265844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.263466 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.265844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.263669 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.265844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.264044 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.265844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.265067 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.267620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.266682 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "kube-api-access-zz6jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:19.267620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.266856 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.267620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.267341 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:19.268825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.268219 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:19.268998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.268959 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.269164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.269133 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.269904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.269875 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:19.273442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.273418 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:19.275179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.275129 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:19.277659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.275865 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.279732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.279703 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc" pod "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" (UID: "6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0"). InnerVolumeSpecName "kube-api-access-rlmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:19.279857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.279834 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy" pod "74083906-8df2-4339-9148-e0d1f5c4de69" (UID: "74083906-8df2-4339-9148-e0d1f5c4de69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.287257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.286913 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:24:19.386620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.334333 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.386620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.334475 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.386620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.334524 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.386620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.381178 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "kube-api-access-czvcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:19.397707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.393648 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:19.397707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.395931 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.454098 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.454215 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.454250 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.454293 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.454351 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.454430 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs\") pod \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\" (UID: \"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d\") " Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455239 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455263 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455279 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455294 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74083906-8df2-4339-9148-e0d1f5c4de69-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455309 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz6jk\" (UniqueName: \"kubernetes.io/projected/74083906-8df2-4339-9148-e0d1f5c4de69-kube-api-access-zz6jk\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455323 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455338 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455352 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455366 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455383 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455397 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455411 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74083906-8df2-4339-9148-e0d1f5c4de69-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455425 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455438 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455453 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455467 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlmdc\" (UniqueName: \"kubernetes.io/projected/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-kube-api-access-rlmdc\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455481 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455495 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455509 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455523 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czvcj\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-kube-api-access-czvcj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.455769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.455539 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74083906-8df2-4339-9148-e0d1f5c4de69-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.457648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.457289 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.458342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.458059 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.463611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.463530 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.468210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.463768 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:19.468210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.464149 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:19.472506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.472480 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token" pod "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" (UID: "e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:19.557990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.557862 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.557990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.557901 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.557990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.557915 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.557990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.557929 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.557990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.557943 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.557990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.557959 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:19.657177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.655927 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst"] Oct 31 23:24:19.659902 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:19.659866 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4c5b40_69d4_4d04_ae24_1709b4e0896f.slice/crio-d5d4211c8fe484f0d078644de1cc0dd6b5f89cddf0157486a66ce7ef8edb4d20 WatchSource:0}: Error finding container d5d4211c8fe484f0d078644de1cc0dd6b5f89cddf0157486a66ce7ef8edb4d20: Status 404 returned error can't find the container with id d5d4211c8fe484f0d078644de1cc0dd6b5f89cddf0157486a66ce7ef8edb4d20 Oct 31 23:24:19.663840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.663783 5043 generic.go:356] "Generic (PLEG): container finished" podID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerID="b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04" exitCode=0 Oct 31 23:24:19.664023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.664005 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" Oct 31 23:24:19.670785 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.670753 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" event={"ID":"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d","Type":"ContainerDied","Data":"b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04"} Oct 31 23:24:19.671005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.670985 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt" event={"ID":"e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d","Type":"ContainerDied","Data":"8cbe78712160dda735fe6b2c019dd0eadc6527b4c974abddfbc86dc019866366"} Oct 31 23:24:19.672380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.672208 5043 scope.go:122] "RemoveContainer" containerID="b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04" Oct 31 23:24:19.680461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.680427 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" event={"ID":"6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0","Type":"ContainerDied","Data":"1f90712791ccc763bc2db1a070f48d307a008450df9dfd22e6b894ebee72a73d"} Oct 31 23:24:19.683492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.683133 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv" Oct 31 23:24:19.700821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.700772 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g"] Oct 31 23:24:19.701180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.701151 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" podUID="7b76e285-2b19-4a24-8c4b-fcba4ee82775" containerName="maas-api" containerID="cri-o://80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a" gracePeriod=30 Oct 31 23:24:19.710253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.710039 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" event={"ID":"74083906-8df2-4339-9148-e0d1f5c4de69","Type":"ContainerDied","Data":"ae73db52c17d95f898b8eaf91c9a5714a27791b9a25c04c1fd5bfbf9260d09f3"} Oct 31 23:24:19.710253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.710150 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr" Oct 31 23:24:19.780015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.779983 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt"] Oct 31 23:24:19.785121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.785091 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-49dvt"] Oct 31 23:24:19.786189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.786047 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:19.786517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.786495 5043 scope.go:122] "RemoveContainer" containerID="b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04" Oct 31 23:24:19.787090 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:19.787061 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04\": container with ID starting with b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04 not found: ID does not exist" containerID="b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04" Oct 31 23:24:19.787347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.787319 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04"} err="failed to get container status \"b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04\": rpc error: code = NotFound desc = could not find container \"b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04\": container with ID starting with b33d5282006386bee744fd24d4cc5cc7a22584d977efeabe25a22f69351bac04 not found: ID does not exist" Oct 31 23:24:19.787478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.787465 5043 scope.go:122] "RemoveContainer" containerID="48dcced70c7a43a399e0827691f5be6beb8a7434633296cbc8c2964b64eacce5" Oct 31 23:24:19.790764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.790743 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv"] Oct 31 23:24:19.794677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.794650 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a1deb6-openshift-default-744454789f-bmcsv"] Oct 31 23:24:19.872830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.872722 5043 scope.go:122] "RemoveContainer" containerID="4211256ebe40147676ce3dc6bd532c51a736f550e2e6f98e987f48f8c8b361e7" Oct 31 23:24:19.908537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:19.908502 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr"] Oct 31 23:24:20.019899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.017536 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" path="/var/lib/kubelet/pods/6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0/volumes" Oct 31 23:24:20.048832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.025629 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" path="/var/lib/kubelet/pods/e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d/volumes" Oct 31 23:24:20.084798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.084723 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:20.148711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.122153 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-2126ff-openshift-default-7f8cf4b68c-chjcr"] Oct 31 23:24:20.189643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.189608 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd"] Oct 31 23:24:20.193581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.193540 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-836c99-n4hjd"] Oct 31 23:24:20.300099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.300007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:20.397101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.397073 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:24:20.456525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.456489 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:20.505952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.505920 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c\" (UniqueName: \"kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c\") pod \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " Oct 31 23:24:20.506132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.506002 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca\") pod \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " Oct 31 23:24:20.506132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.506116 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls\") pod \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " Oct 31 23:24:20.506305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.506286 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs\") pod \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\" (UID: \"7b76e285-2b19-4a24-8c4b-fcba4ee82775\") " Oct 31 23:24:20.506401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.506343 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca" pod "7b76e285-2b19-4a24-8c4b-fcba4ee82775" (UID: "7b76e285-2b19-4a24-8c4b-fcba4ee82775"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:20.506963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.506939 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b76e285-2b19-4a24-8c4b-fcba4ee82775-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:20.508332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.508307 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls" pod "7b76e285-2b19-4a24-8c4b-fcba4ee82775" (UID: "7b76e285-2b19-4a24-8c4b-fcba4ee82775"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:20.508332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.508323 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs" pod "7b76e285-2b19-4a24-8c4b-fcba4ee82775" (UID: "7b76e285-2b19-4a24-8c4b-fcba4ee82775"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:20.508629 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.508609 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c" pod "7b76e285-2b19-4a24-8c4b-fcba4ee82775" (UID: "7b76e285-2b19-4a24-8c4b-fcba4ee82775"). InnerVolumeSpecName "kube-api-access-j9f9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:20.609958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.609868 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:20.610162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.610144 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b76e285-2b19-4a24-8c4b-fcba4ee82775-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:20.610257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.610245 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9f9c\" (UniqueName: \"kubernetes.io/projected/7b76e285-2b19-4a24-8c4b-fcba4ee82775-kube-api-access-j9f9c\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:20.707237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.707195 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:20.872376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.872281 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" event={"ID":"3c4c5b40-69d4-4d04-ae24-1709b4e0896f","Type":"ContainerStarted","Data":"56a2ddc34092de7d21ecdcb1115e783d235e7e479f8c99a4148a5dd96d00194a"} Oct 31 23:24:20.872376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.872370 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" event={"ID":"3c4c5b40-69d4-4d04-ae24-1709b4e0896f","Type":"ContainerStarted","Data":"d5d4211c8fe484f0d078644de1cc0dd6b5f89cddf0157486a66ce7ef8edb4d20"} Oct 31 23:24:20.875332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.875297 5043 generic.go:356] "Generic (PLEG): container finished" podID="7b76e285-2b19-4a24-8c4b-fcba4ee82775" containerID="80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a" exitCode=0 Oct 31 23:24:20.875510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.875372 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" event={"ID":"7b76e285-2b19-4a24-8c4b-fcba4ee82775","Type":"ContainerDied","Data":"80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a"} Oct 31 23:24:20.875510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.875454 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" event={"ID":"7b76e285-2b19-4a24-8c4b-fcba4ee82775","Type":"ContainerDied","Data":"1174bdf6334903bf1cca422fefd03c777d6f0abdd2eda10e2d3ddd726aa767eb"} Oct 31 23:24:20.875510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.875487 5043 scope.go:122] "RemoveContainer" containerID="80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a" Oct 31 23:24:20.875859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.875826 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g" Oct 31 23:24:20.898409 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.898381 5043 scope.go:122] "RemoveContainer" containerID="80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a" Oct 31 23:24:20.898821 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:20.898802 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a\": container with ID starting with 80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a not found: ID does not exist" containerID="80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a" Oct 31 23:24:20.898945 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.898927 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a"} err="failed to get container status \"80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a\": rpc error: code = NotFound desc = could not find container \"80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a\": container with ID starting with 80749f1818329cf92a04815114610f0818459b224146e67b837afe87d743358a not found: ID does not exist" Oct 31 23:24:20.899637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.899584 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" podStartSLOduration=2.8995668610000003 podStartE2EDuration="2.899566861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:20.891569688 +0000 UTC m=+2087.121124101" watchObservedRunningTime="2026-10-31 23:24:20.899566861 +0000 UTC m=+2087.129121273" Oct 31 23:24:20.956236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.956196 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g"] Oct 31 23:24:20.963116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:20.963083 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-836c99-57b859b6c4-2ks6g"] Oct 31 23:24:21.136664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.136002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:21.288066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.288029 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:21.464773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.462248 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:21.511732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.511685 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw"] Oct 31 23:24:21.574379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.574341 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d6685e-4sxqw"] Oct 31 23:24:21.676951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.676913 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:21.834569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.822613 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" containerID="cri-o://73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d" gracePeriod=25 Oct 31 23:24:21.888626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.888596 5043 generic.go:356] "Generic (PLEG): container finished" podID="3c4c5b40-69d4-4d04-ae24-1709b4e0896f" containerID="56a2ddc34092de7d21ecdcb1115e783d235e7e479f8c99a4148a5dd96d00194a" exitCode=0 Oct 31 23:24:21.888804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.888676 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" event={"ID":"3c4c5b40-69d4-4d04-ae24-1709b4e0896f","Type":"ContainerDied","Data":"56a2ddc34092de7d21ecdcb1115e783d235e7e479f8c99a4148a5dd96d00194a"} Oct 31 23:24:21.976781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.973463 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad03db6-9f68-4499-abea-aa85bc145b09" path="/var/lib/kubelet/pods/5ad03db6-9f68-4499-abea-aa85bc145b09/volumes" Oct 31 23:24:21.976781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.974170 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" path="/var/lib/kubelet/pods/74083906-8df2-4339-9148-e0d1f5c4de69/volumes" Oct 31 23:24:21.976781 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.974764 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b76e285-2b19-4a24-8c4b-fcba4ee82775" path="/var/lib/kubelet/pods/7b76e285-2b19-4a24-8c4b-fcba4ee82775/volumes" Oct 31 23:24:21.978003 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.977530 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcda8e48-b494-4fd1-b435-4780fe793cce" path="/var/lib/kubelet/pods/fcda8e48-b494-4fd1-b435-4780fe793cce/volumes" Oct 31 23:24:21.978630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:21.978602 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:22.193539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.193513 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:24:22.305489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.305458 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume\") pod \"f852be0e-4036-4565-a1c1-d20c99495370\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " Oct 31 23:24:22.305839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.305817 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9\" (UniqueName: \"kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9\") pod \"f852be0e-4036-4565-a1c1-d20c99495370\" (UID: \"f852be0e-4036-4565-a1c1-d20c99495370\") " Oct 31 23:24:22.305905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.305887 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume" pod "f852be0e-4036-4565-a1c1-d20c99495370" (UID: "f852be0e-4036-4565-a1c1-d20c99495370"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:22.306361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.306340 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f852be0e-4036-4565-a1c1-d20c99495370-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:22.307983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.307956 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9" pod "f852be0e-4036-4565-a1c1-d20c99495370" (UID: "f852be0e-4036-4565-a1c1-d20c99495370"). InnerVolumeSpecName "kube-api-access-g8wn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:22.408078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.408044 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wn9\" (UniqueName: \"kubernetes.io/projected/f852be0e-4036-4565-a1c1-d20c99495370-kube-api-access-g8wn9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:22.624606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.620973 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:22.762962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.762925 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:22.899480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.899392 5043 generic.go:356] "Generic (PLEG): container finished" podID="f852be0e-4036-4565-a1c1-d20c99495370" containerID="73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d" exitCode=0 Oct 31 23:24:22.899480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.899429 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" Oct 31 23:24:22.899480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.899468 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" event={"ID":"f852be0e-4036-4565-a1c1-d20c99495370","Type":"ContainerDied","Data":"73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d"} Oct 31 23:24:22.900053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.899522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4" event={"ID":"f852be0e-4036-4565-a1c1-d20c99495370","Type":"ContainerDied","Data":"cf89b18638723a639d5415fdedc8232dd38b3c25e39b2a84fcb66eeb81e2f79c"} Oct 31 23:24:22.900053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.899589 5043 scope.go:122] "RemoveContainer" containerID="73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d" Oct 31 23:24:22.921499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.921475 5043 scope.go:122] "RemoveContainer" containerID="73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d" Oct 31 23:24:22.922164 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:22.922139 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d\": container with ID starting with 73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d not found: ID does not exist" containerID="73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d" Oct 31 23:24:22.940116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.940072 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d"} err="failed to get container status \"73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d\": rpc error: code = NotFound desc = could not find container \"73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d\": container with ID starting with 73554e00ecd0a22b3041696bb86fe931d7d934c26244e54d16f742764604cf8d not found: ID does not exist" Oct 31 23:24:22.940321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.940301 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4"] Oct 31 23:24:22.945462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:22.945434 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f80dfa-5c6bbbbdb4-x9qw4"] Oct 31 23:24:23.126738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.126706 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:23.249226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.249188 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj\") pod \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " Oct 31 23:24:23.249417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.249273 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca\") pod \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " Oct 31 23:24:23.249636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.249604 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca\") pod \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\" (UID: \"3c4c5b40-69d4-4d04-ae24-1709b4e0896f\") " Oct 31 23:24:23.250215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.250185 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca" pod "3c4c5b40-69d4-4d04-ae24-1709b4e0896f" (UID: "3c4c5b40-69d4-4d04-ae24-1709b4e0896f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:23.251624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.251599 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca" pod "3c4c5b40-69d4-4d04-ae24-1709b4e0896f" (UID: "3c4c5b40-69d4-4d04-ae24-1709b4e0896f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:23.252037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.252012 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj" pod "3c4c5b40-69d4-4d04-ae24-1709b4e0896f" (UID: "3c4c5b40-69d4-4d04-ae24-1709b4e0896f"). InnerVolumeSpecName "kube-api-access-qqkzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:23.352191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.352150 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-kube-api-access-qqkzj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:23.352191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.352192 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:23.352446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.352209 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c4c5b40-69d4-4d04-ae24-1709b4e0896f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:23.422164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.422123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:23.916023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.912405 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" Oct 31 23:24:23.916023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.912435 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst" event={"ID":"3c4c5b40-69d4-4d04-ae24-1709b4e0896f","Type":"ContainerDied","Data":"d5d4211c8fe484f0d078644de1cc0dd6b5f89cddf0157486a66ce7ef8edb4d20"} Oct 31 23:24:23.916023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.912524 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d4211c8fe484f0d078644de1cc0dd6b5f89cddf0157486a66ce7ef8edb4d20" Oct 31 23:24:23.968851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.968798 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f852be0e-4036-4565-a1c1-d20c99495370" path="/var/lib/kubelet/pods/f852be0e-4036-4565-a1c1-d20c99495370/volumes" Oct 31 23:24:23.970677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:23.970645 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp"] Oct 31 23:24:24.029969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:24.029930 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:24.196077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:24.195983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:24.745314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:24.744756 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:25.460075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:25.460035 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:25.758973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:25.758703 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:25.958958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:25.958891 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:26.129320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.129230 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:26.302568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.298118 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:26.657940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.657899 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g"] Oct 31 23:24:26.658953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.658930 5043 cpu_manager.go:414] "containerMap: removing container" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" Oct 31 23:24:26.658953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.658955 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.658977 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.658985 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659003 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7b76e285-2b19-4a24-8c4b-fcba4ee82775" containerName="maas-api" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659011 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b76e285-2b19-4a24-8c4b-fcba4ee82775" containerName="maas-api" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659028 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659035 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659080 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659087 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659104 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659112 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659124 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3c4c5b40-69d4-4d04-ae24-1709b4e0896f" containerName="revoke-keys" Oct 31 23:24:26.659133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659132 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4c5b40-69d4-4d04-ae24-1709b4e0896f" containerName="revoke-keys" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659281 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659293 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659306 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b76e285-2b19-4a24-8c4b-fcba4ee82775" containerName="maas-api" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659315 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e703fb61-c4ed-4b10-95e0-cdec8d4b6c1d" containerName="istio-proxy" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659331 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659341 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe93bc4-f2d7-4543-a75a-da1bb9e18cb0" containerName="istio-proxy" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659351 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f852be0e-4036-4565-a1c1-d20c99495370" containerName="payload-processing" Oct 31 23:24:26.659765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.659362 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4c5b40-69d4-4d04-ae24-1709b4e0896f" containerName="revoke-keys" Oct 31 23:24:26.692567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.692512 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g"] Oct 31 23:24:26.692733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.692664 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.696797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.696775 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:24:26.698066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.698046 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:24:26.721360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.720952 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:26.721360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.721089 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:26.826777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.826735 5043 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:24:26.826971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.826821 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:24:26.841681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.841639 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.841883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.841853 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.841974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.841957 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwpm\" (UniqueName: \"kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.944118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.944023 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.944317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.944184 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.944413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.944390 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwpm\" (UniqueName: \"kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.950283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.950249 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.952104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.952077 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:26.971832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:26.971809 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwpm\" (UniqueName: \"kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:27.031857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:27.031808 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:27.120028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:27.113372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:27.563631 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:27.563599 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85c9373_7a8b_4cb7_919a_d0f9f451b383.slice/crio-2f4779962f95aea3366801321e512fd21849c8585b4d535e4bb6b02ece246a41 WatchSource:0}: Error finding container 2f4779962f95aea3366801321e512fd21849c8585b4d535e4bb6b02ece246a41: Status 404 returned error can't find the container with id 2f4779962f95aea3366801321e512fd21849c8585b4d535e4bb6b02ece246a41 Oct 31 23:24:27.569376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:27.568303 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g"] Oct 31 23:24:28.026589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:27.972331 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" event={"ID":"f85c9373-7a8b-4cb7-919a-d0f9f451b383","Type":"ContainerStarted","Data":"4209a629a4c3b60feb396533d4e6097b4f625f6d9cca941c5ad90dc43424dca1"} Oct 31 23:24:28.026589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:27.972381 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" event={"ID":"f85c9373-7a8b-4cb7-919a-d0f9f451b383","Type":"ContainerStarted","Data":"2f4779962f95aea3366801321e512fd21849c8585b4d535e4bb6b02ece246a41"} Oct 31 23:24:28.026589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:27.996575 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" podStartSLOduration=1.996562139 podStartE2EDuration="1.996562139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:27.99577837 +0000 UTC m=+2094.225332780" watchObservedRunningTime="2026-10-31 23:24:27.996562139 +0000 UTC m=+2094.226116543" Oct 31 23:24:28.088107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.083806 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:28.807986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.807950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:28.817252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.817222 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx"] Oct 31 23:24:28.952729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.952693 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:28.972049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.972019 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" podUID="fb57d425-439d-4b90-aa22-11d8960fc37a" containerName="payload-pre-processing" containerID="cri-o://c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d" gracePeriod=25 Oct 31 23:24:28.985099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.985072 5043 generic.go:356] "Generic (PLEG): container finished" podID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerID="b906f09b982eaecdf92a12fafa3a52c67ac50d1c190b91475c330fa7890d54e0" exitCode=0 Oct 31 23:24:28.985239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.985152 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" event={"ID":"5f5396c0-d04b-4150-9ad9-179d7319cdb4","Type":"ContainerDied","Data":"b906f09b982eaecdf92a12fafa3a52c67ac50d1c190b91475c330fa7890d54e0"} Oct 31 23:24:28.985643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.985624 5043 scope.go:122] "RemoveContainer" containerID="b906f09b982eaecdf92a12fafa3a52c67ac50d1c190b91475c330fa7890d54e0" Oct 31 23:24:28.987059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.987038 5043 generic.go:356] "Generic (PLEG): container finished" podID="f85c9373-7a8b-4cb7-919a-d0f9f451b383" containerID="4209a629a4c3b60feb396533d4e6097b4f625f6d9cca941c5ad90dc43424dca1" exitCode=0 Oct 31 23:24:28.987163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.987122 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" event={"ID":"f85c9373-7a8b-4cb7-919a-d0f9f451b383","Type":"ContainerDied","Data":"4209a629a4c3b60feb396533d4e6097b4f625f6d9cca941c5ad90dc43424dca1"} Oct 31 23:24:28.987346 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.987322 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:24:28.987399 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.987387 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:24:28.987435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:28.987420 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:24:29.085370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.085322 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f"] Oct 31 23:24:29.089224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.086824 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" containerID="cri-o://68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5" gracePeriod=30 Oct 31 23:24:29.094882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.094855 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h"] Oct 31 23:24:29.095080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.095055 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" containerID="cri-o://0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc" gracePeriod=30 Oct 31 23:24:29.163747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.163719 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:29.185665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.184658 5043 patch_prober.go:29] interesting pod/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:24:29.185665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.184712 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:24:29.651612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.648347 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:29.651612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.648823 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:24:29.820625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.803531 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume\") pod \"fb57d425-439d-4b90-aa22-11d8960fc37a\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " Oct 31 23:24:29.852874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.845431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:29.853200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.848913 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume" pod "fb57d425-439d-4b90-aa22-11d8960fc37a" (UID: "fb57d425-439d-4b90-aa22-11d8960fc37a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:29.854003 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.853979 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk\" (UniqueName: \"kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk\") pod \"fb57d425-439d-4b90-aa22-11d8960fc37a\" (UID: \"fb57d425-439d-4b90-aa22-11d8960fc37a\") " Oct 31 23:24:29.856082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.856041 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb57d425-439d-4b90-aa22-11d8960fc37a-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:29.861216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.861182 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk" pod "fb57d425-439d-4b90-aa22-11d8960fc37a" (UID: "fb57d425-439d-4b90-aa22-11d8960fc37a"). InnerVolumeSpecName "kube-api-access-9k5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:29.921206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.905670 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6"] Oct 31 23:24:29.921206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.906870 5043 cpu_manager.go:414] "containerMap: removing container" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" Oct 31 23:24:29.921206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.906894 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74083906-8df2-4339-9148-e0d1f5c4de69" containerName="istio-proxy" Oct 31 23:24:29.921206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.906957 5043 cpu_manager.go:414] "containerMap: removing container" podUID="fb57d425-439d-4b90-aa22-11d8960fc37a" containerName="payload-pre-processing" Oct 31 23:24:29.921206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.906967 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb57d425-439d-4b90-aa22-11d8960fc37a" containerName="payload-pre-processing" Oct 31 23:24:29.921206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.907364 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb57d425-439d-4b90-aa22-11d8960fc37a" containerName="payload-pre-processing" Oct 31 23:24:29.944508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.940315 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6"] Oct 31 23:24:29.953647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.949753 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:29.961273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.960842 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:29.961273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.960977 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t585s\" (UniqueName: \"kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:29.961273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.961070 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:29.961273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.961151 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k5bk\" (UniqueName: \"kubernetes.io/projected/fb57d425-439d-4b90-aa22-11d8960fc37a-kube-api-access-9k5bk\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:29.978575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:29.978087 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8"] Oct 31 23:24:30.005899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.005865 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:30.020126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.020100 5043 generic.go:356] "Generic (PLEG): container finished" podID="fb57d425-439d-4b90-aa22-11d8960fc37a" containerID="c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d" exitCode=0 Oct 31 23:24:30.020260 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.020178 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" event={"ID":"fb57d425-439d-4b90-aa22-11d8960fc37a","Type":"ContainerDied","Data":"c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d"} Oct 31 23:24:30.020260 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.020239 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" event={"ID":"fb57d425-439d-4b90-aa22-11d8960fc37a","Type":"ContainerDied","Data":"102a53b5e701ec28a4c10034f5d830f7d8d8e54851fa8a22fb22fec936c48813"} Oct 31 23:24:30.020260 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.020259 5043 scope.go:122] "RemoveContainer" containerID="c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d" Oct 31 23:24:30.020421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.020365 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp" Oct 31 23:24:30.023815 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.023789 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" event={"ID":"5f5396c0-d04b-4150-9ad9-179d7319cdb4","Type":"ContainerStarted","Data":"9cc3a7985fb08998e16e68e50fad07e3c317d5287fb7b894f1ca8f689c80da4d"} Oct 31 23:24:30.023923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.023870 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" containerID="cri-o://9cc3a7985fb08998e16e68e50fad07e3c317d5287fb7b894f1ca8f689c80da4d" gracePeriod=30 Oct 31 23:24:30.059819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.053997 5043 scope.go:122] "RemoveContainer" containerID="c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d" Oct 31 23:24:30.059819 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:30.055317 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d\": container with ID starting with c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d not found: ID does not exist" containerID="c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d" Oct 31 23:24:30.059819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.055357 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d"} err="failed to get container status \"c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d\": rpc error: code = NotFound desc = could not find container \"c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d\": container with ID starting with c64c10f1ce010a366133500e918928cc227a34ecb46321aa4f4cdde1e8fd6d7d not found: ID does not exist" Oct 31 23:24:30.111635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.094909 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.111635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.095100 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t585s\" (UniqueName: \"kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.111635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.095231 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.111635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.102748 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.165582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.163066 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.169574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.167274 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp"] Oct 31 23:24:30.188585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.172330 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t585s\" (UniqueName: \"kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s\") pod \"maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.188585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.187916 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf279c-7cc8c475d7-xgqqp"] Oct 31 23:24:30.284068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.284034 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:30.384167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.383864 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:30.500038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.499986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:30.516233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.516199 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca\") pod \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " Oct 31 23:24:30.516390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.516302 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm\" (UniqueName: \"kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm\") pod \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " Oct 31 23:24:30.516462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.516389 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca\") pod \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\" (UID: \"f85c9373-7a8b-4cb7-919a-d0f9f451b383\") " Oct 31 23:24:30.516853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.516820 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca" pod "f85c9373-7a8b-4cb7-919a-d0f9f451b383" (UID: "f85c9373-7a8b-4cb7-919a-d0f9f451b383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:30.517126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.517108 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:30.518935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.518909 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca" pod "f85c9373-7a8b-4cb7-919a-d0f9f451b383" (UID: "f85c9373-7a8b-4cb7-919a-d0f9f451b383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:30.520338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.520309 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm" pod "f85c9373-7a8b-4cb7-919a-d0f9f451b383" (UID: "f85c9373-7a8b-4cb7-919a-d0f9f451b383"). InnerVolumeSpecName "kube-api-access-4kwpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:30.591618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.591595 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6"] Oct 31 23:24:30.592705 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:30.592679 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53ab05c_86a3_42da_94ef_c520e1227ae8.slice/crio-8ad53cfa677e2cd26d5f390080db099edce137e892af78a090c85bf9842aa0d3 WatchSource:0}: Error finding container 8ad53cfa677e2cd26d5f390080db099edce137e892af78a090c85bf9842aa0d3: Status 404 returned error can't find the container with id 8ad53cfa677e2cd26d5f390080db099edce137e892af78a090c85bf9842aa0d3 Oct 31 23:24:30.618622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.618454 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f85c9373-7a8b-4cb7-919a-d0f9f451b383-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:30.618622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.618601 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwpm\" (UniqueName: \"kubernetes.io/projected/f85c9373-7a8b-4cb7-919a-d0f9f451b383-kube-api-access-4kwpm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:30.681219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.681190 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm"] Oct 31 23:24:30.681978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.681960 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f85c9373-7a8b-4cb7-919a-d0f9f451b383" containerName="revoke-keys" Oct 31 23:24:30.682044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.681978 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85c9373-7a8b-4cb7-919a-d0f9f451b383" containerName="revoke-keys" Oct 31 23:24:30.682122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.682113 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85c9373-7a8b-4cb7-919a-d0f9f451b383" containerName="revoke-keys" Oct 31 23:24:30.688245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.687974 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.695237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.695216 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm"] Oct 31 23:24:30.822085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.822049 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.822085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.822088 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gr6\" (UniqueName: \"kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.822309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.822290 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.924319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.924292 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.924558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.924515 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gr6\" (UniqueName: \"kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.924836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.924801 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.925384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.925357 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.927296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.927264 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:30.937036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:30.937014 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gr6\" (UniqueName: \"kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6\") pod \"maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:31.018462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.018370 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:31.050392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.050345 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:31.060509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.060482 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" Oct 31 23:24:31.060859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.060491 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g" event={"ID":"f85c9373-7a8b-4cb7-919a-d0f9f451b383","Type":"ContainerDied","Data":"2f4779962f95aea3366801321e512fd21849c8585b4d535e4bb6b02ece246a41"} Oct 31 23:24:31.060956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.060889 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f4779962f95aea3366801321e512fd21849c8585b4d535e4bb6b02ece246a41" Oct 31 23:24:31.062292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.062259 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" event={"ID":"b53ab05c-86a3-42da-94ef-c520e1227ae8","Type":"ContainerStarted","Data":"8ad53cfa677e2cd26d5f390080db099edce137e892af78a090c85bf9842aa0d3"} Oct 31 23:24:31.401998 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:31.401965 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf60d0ba_d83d_47bc_b107_7b72e664e26e.slice/crio-f59b5f129cc8ce6f633dce15b4a00febd99514ac216fcd6cac28fd1ebc6f96ef WatchSource:0}: Error finding container f59b5f129cc8ce6f633dce15b4a00febd99514ac216fcd6cac28fd1ebc6f96ef: Status 404 returned error can't find the container with id f59b5f129cc8ce6f633dce15b4a00febd99514ac216fcd6cac28fd1ebc6f96ef Oct 31 23:24:31.407762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.407166 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm"] Oct 31 23:24:31.937335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:31.937301 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb57d425-439d-4b90-aa22-11d8960fc37a" path="/var/lib/kubelet/pods/fb57d425-439d-4b90-aa22-11d8960fc37a/volumes" Oct 31 23:24:32.074472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.074427 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" event={"ID":"df60d0ba-d83d-47bc-b107-7b72e664e26e","Type":"ContainerStarted","Data":"ada1dcc7626b29492ca802e53f90568ada509f47d91a9179c1e9035e38cb87eb"} Oct 31 23:24:32.074673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.074496 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" event={"ID":"df60d0ba-d83d-47bc-b107-7b72e664e26e","Type":"ContainerStarted","Data":"f59b5f129cc8ce6f633dce15b4a00febd99514ac216fcd6cac28fd1ebc6f96ef"} Oct 31 23:24:32.076246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.076217 5043 generic.go:356] "Generic (PLEG): container finished" podID="b53ab05c-86a3-42da-94ef-c520e1227ae8" containerID="747eff1b03fa7c094df4b48984459a803e014769da6a6ecccc30a1090fa6228d" exitCode=0 Oct 31 23:24:32.076374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.076255 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" event={"ID":"b53ab05c-86a3-42da-94ef-c520e1227ae8","Type":"ContainerDied","Data":"747eff1b03fa7c094df4b48984459a803e014769da6a6ecccc30a1090fa6228d"} Oct 31 23:24:32.096401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.096354 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" podStartSLOduration=2.09634128 podStartE2EDuration="2.09634128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:32.092797838 +0000 UTC m=+2098.322352250" watchObservedRunningTime="2026-10-31 23:24:32.09634128 +0000 UTC m=+2098.325895691" Oct 31 23:24:32.145070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.145011 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:32.785092 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:32.785056 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:33.088430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.088342 5043 generic.go:356] "Generic (PLEG): container finished" podID="df60d0ba-d83d-47bc-b107-7b72e664e26e" containerID="ada1dcc7626b29492ca802e53f90568ada509f47d91a9179c1e9035e38cb87eb" exitCode=0 Oct 31 23:24:33.088647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.088420 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" event={"ID":"df60d0ba-d83d-47bc-b107-7b72e664e26e","Type":"ContainerDied","Data":"ada1dcc7626b29492ca802e53f90568ada509f47d91a9179c1e9035e38cb87eb"} Oct 31 23:24:33.296380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.296352 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:33.376461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.376385 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s\" (UniqueName: \"kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s\") pod \"b53ab05c-86a3-42da-94ef-c520e1227ae8\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " Oct 31 23:24:33.376631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.376613 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca\") pod \"b53ab05c-86a3-42da-94ef-c520e1227ae8\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " Oct 31 23:24:33.376699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.376685 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca\") pod \"b53ab05c-86a3-42da-94ef-c520e1227ae8\" (UID: \"b53ab05c-86a3-42da-94ef-c520e1227ae8\") " Oct 31 23:24:33.376965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.376934 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca" pod "b53ab05c-86a3-42da-94ef-c520e1227ae8" (UID: "b53ab05c-86a3-42da-94ef-c520e1227ae8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:33.377234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.377214 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.378663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.378630 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s" pod "b53ab05c-86a3-42da-94ef-c520e1227ae8" (UID: "b53ab05c-86a3-42da-94ef-c520e1227ae8"). InnerVolumeSpecName "kube-api-access-t585s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:33.378764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.378667 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca" pod "b53ab05c-86a3-42da-94ef-c520e1227ae8" (UID: "b53ab05c-86a3-42da-94ef-c520e1227ae8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:33.479054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.479022 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b53ab05c-86a3-42da-94ef-c520e1227ae8-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.479054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.479051 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t585s\" (UniqueName: \"kubernetes.io/projected/b53ab05c-86a3-42da-94ef-c520e1227ae8-kube-api-access-t585s\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:33.508264 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.508218 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:33.817818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:33.817759 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" containerID="cri-o://b55434f93433197d088c4b7dd6b426fab9a15158df78e6eee9ce851067bddf0e" gracePeriod=25 Oct 31 23:24:34.035071 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:34.033476 5043 info.go:109] Failed to get network devices: open /sys/class/net/82627e76da4027d/address: no such file or directory Oct 31 23:24:34.152350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.145306 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" event={"ID":"b53ab05c-86a3-42da-94ef-c520e1227ae8","Type":"ContainerDied","Data":"8ad53cfa677e2cd26d5f390080db099edce137e892af78a090c85bf9842aa0d3"} Oct 31 23:24:34.152350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.145365 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad53cfa677e2cd26d5f390080db099edce137e892af78a090c85bf9842aa0d3" Oct 31 23:24:34.152350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.145451 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6" Oct 31 23:24:34.152350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.148895 5043 generic.go:356] "Generic (PLEG): container finished" podID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerID="b55434f93433197d088c4b7dd6b426fab9a15158df78e6eee9ce851067bddf0e" exitCode=0 Oct 31 23:24:34.152350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.149172 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" event={"ID":"1d56bf0b-7968-4952-9917-ba0bc5cba955","Type":"ContainerDied","Data":"b55434f93433197d088c4b7dd6b426fab9a15158df78e6eee9ce851067bddf0e"} Oct 31 23:24:34.190740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.188349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:34.241588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.239521 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:24:34.327596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.324200 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume\") pod \"1d56bf0b-7968-4952-9917-ba0bc5cba955\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " Oct 31 23:24:34.327596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.324405 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f\" (UniqueName: \"kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f\") pod \"1d56bf0b-7968-4952-9917-ba0bc5cba955\" (UID: \"1d56bf0b-7968-4952-9917-ba0bc5cba955\") " Oct 31 23:24:34.327596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.326074 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume" pod "1d56bf0b-7968-4952-9917-ba0bc5cba955" (UID: "1d56bf0b-7968-4952-9917-ba0bc5cba955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:34.341047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.340360 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f" pod "1d56bf0b-7968-4952-9917-ba0bc5cba955" (UID: "1d56bf0b-7968-4952-9917-ba0bc5cba955"). InnerVolumeSpecName "kube-api-access-2tp7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.427271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.427235 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tp7f\" (UniqueName: \"kubernetes.io/projected/1d56bf0b-7968-4952-9917-ba0bc5cba955-kube-api-access-2tp7f\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.427402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.427276 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56bf0b-7968-4952-9917-ba0bc5cba955-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.457835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.457801 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:34.485393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.485363 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:24:34.514299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.514270 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:24:34.527930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.527899 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca\") pod \"df60d0ba-d83d-47bc-b107-7b72e664e26e\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " Oct 31 23:24:34.528094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.527971 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6\" (UniqueName: \"kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6\") pod \"df60d0ba-d83d-47bc-b107-7b72e664e26e\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " Oct 31 23:24:34.528206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.528185 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca\") pod \"df60d0ba-d83d-47bc-b107-7b72e664e26e\" (UID: \"df60d0ba-d83d-47bc-b107-7b72e664e26e\") " Oct 31 23:24:34.528275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.528246 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca" pod "df60d0ba-d83d-47bc-b107-7b72e664e26e" (UID: "df60d0ba-d83d-47bc-b107-7b72e664e26e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:34.529098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.529075 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.534934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.534904 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca" pod "df60d0ba-d83d-47bc-b107-7b72e664e26e" (UID: "df60d0ba-d83d-47bc-b107-7b72e664e26e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:34.536651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.536627 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6" pod "df60d0ba-d83d-47bc-b107-7b72e664e26e" (UID: "df60d0ba-d83d-47bc-b107-7b72e664e26e"). InnerVolumeSpecName "kube-api-access-b4gr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.612482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.612442 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv"] Oct 31 23:24:34.629579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629532 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.629774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629623 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.629774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629678 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.629774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629709 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.629774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629774 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.630015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629812 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.630015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629850 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.630015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629883 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.630015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629910 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.630015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629962 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.630015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.629994 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630046 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630086 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket\") pod \"c2797e68-0138-44e2-bc82-71ed4daa86b5\" (UID: \"c2797e68-0138-44e2-bc82-71ed4daa86b5\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630097 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630165 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630216 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630255 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630291 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630300 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630347 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630415 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo\") pod \"a302ec33-70dd-42ae-baa1-aaa48d20a369\" (UID: \"a302ec33-70dd-42ae-baa1-aaa48d20a369\") " Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630610 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630704 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.630803 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631020 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631290 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631311 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df60d0ba-d83d-47bc-b107-7b72e664e26e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631326 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2797e68-0138-44e2-bc82-71ed4daa86b5-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631355 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631369 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631371 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631383 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631399 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631413 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.631428 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gr6\" (UniqueName: \"kubernetes.io/projected/df60d0ba-d83d-47bc-b107-7b72e664e26e-kube-api-access-b4gr6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.632014 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.636522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.636049 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.638601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.636872 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.638601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.637095 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:34.638601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.637683 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:34.638601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.637713 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "kube-api-access-46mgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.638601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.637996 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "kube-api-access-cp8nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.639676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.639209 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:34.640911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.640889 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token" pod "a302ec33-70dd-42ae-baa1-aaa48d20a369" (UID: "a302ec33-70dd-42ae-baa1-aaa48d20a369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.642437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.642415 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token" pod "c2797e68-0138-44e2-bc82-71ed4daa86b5" (UID: "c2797e68-0138-44e2-bc82-71ed4daa86b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:34.734709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734611 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mgk\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-kube-api-access-46mgk\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.734709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734653 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.734709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734675 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a302ec33-70dd-42ae-baa1-aaa48d20a369-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.734709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734689 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.734709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734703 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.735133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734716 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.735133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734752 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8nw\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-kube-api-access-cp8nw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.735133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734771 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.735133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734787 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.735133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734802 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a302ec33-70dd-42ae-baa1-aaa48d20a369-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.735133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.734816 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2797e68-0138-44e2-bc82-71ed4daa86b5-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:34.738940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.738904 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:34.975672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:34.975440 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:35.163564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.161890 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" Oct 31 23:24:35.163564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.161928 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm" event={"ID":"df60d0ba-d83d-47bc-b107-7b72e664e26e","Type":"ContainerDied","Data":"f59b5f129cc8ce6f633dce15b4a00febd99514ac216fcd6cac28fd1ebc6f96ef"} Oct 31 23:24:35.163564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.161977 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f59b5f129cc8ce6f633dce15b4a00febd99514ac216fcd6cac28fd1ebc6f96ef" Oct 31 23:24:35.164511 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.164479 5043 generic.go:356] "Generic (PLEG): container finished" podID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerID="9cc3a7985fb08998e16e68e50fad07e3c317d5287fb7b894f1ca8f689c80da4d" exitCode=0 Oct 31 23:24:35.164666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.164598 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" event={"ID":"5f5396c0-d04b-4150-9ad9-179d7319cdb4","Type":"ContainerDied","Data":"9cc3a7985fb08998e16e68e50fad07e3c317d5287fb7b894f1ca8f689c80da4d"} Oct 31 23:24:35.164712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.164681 5043 scope.go:122] "RemoveContainer" containerID="b906f09b982eaecdf92a12fafa3a52c67ac50d1c190b91475c330fa7890d54e0" Oct 31 23:24:35.166973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.166947 5043 generic.go:356] "Generic (PLEG): container finished" podID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerID="0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc" exitCode=0 Oct 31 23:24:35.167091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.167003 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" Oct 31 23:24:35.167091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.167045 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" event={"ID":"a302ec33-70dd-42ae-baa1-aaa48d20a369","Type":"ContainerDied","Data":"0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc"} Oct 31 23:24:35.167203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.167145 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h" event={"ID":"a302ec33-70dd-42ae-baa1-aaa48d20a369","Type":"ContainerDied","Data":"33b73166b543624a7037bba9edfa3eba70da968a9b240a451f91ac96ef87b2cb"} Oct 31 23:24:35.170857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.170837 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" Oct 31 23:24:35.170985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.170839 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx" event={"ID":"1d56bf0b-7968-4952-9917-ba0bc5cba955","Type":"ContainerDied","Data":"82627e76da4027d3a45d66ec6de7e4e8a7547632272d3a88a6530c1ec4a568a4"} Oct 31 23:24:35.173225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.173197 5043 generic.go:356] "Generic (PLEG): container finished" podID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerID="68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5" exitCode=0 Oct 31 23:24:35.173318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.173256 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" event={"ID":"c2797e68-0138-44e2-bc82-71ed4daa86b5","Type":"ContainerDied","Data":"68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5"} Oct 31 23:24:35.173318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.173303 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" event={"ID":"c2797e68-0138-44e2-bc82-71ed4daa86b5","Type":"ContainerDied","Data":"bfcb72e1386b5791af1c359ad70df36adba4fa07b0194042637a9dde2bc73ba2"} Oct 31 23:24:35.173446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.173371 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f" Oct 31 23:24:35.189615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.189584 5043 scope.go:122] "RemoveContainer" containerID="0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc" Oct 31 23:24:35.230289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.230222 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f"] Oct 31 23:24:35.234727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.234701 5043 scope.go:122] "RemoveContainer" containerID="ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50" Oct 31 23:24:35.238583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.238519 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-phs6f"] Oct 31 23:24:35.252160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.252129 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx"] Oct 31 23:24:35.255232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.255212 5043 scope.go:122] "RemoveContainer" containerID="0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc" Oct 31 23:24:35.255626 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:35.255596 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc\": container with ID starting with 0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc not found: ID does not exist" containerID="0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc" Oct 31 23:24:35.255725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.255636 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc"} err="failed to get container status \"0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc\": rpc error: code = NotFound desc = could not find container \"0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc\": container with ID starting with 0704e2a955123cb394ed1276d968c3db5ec63bbda47befe4d850db6cb13328fc not found: ID does not exist" Oct 31 23:24:35.255725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.255660 5043 scope.go:122] "RemoveContainer" containerID="ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50" Oct 31 23:24:35.261074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.261042 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-270e8c-9d8466bf5-nsznx"] Oct 31 23:24:35.262307 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:35.261758 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50\": container with ID starting with ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50 not found: ID does not exist" containerID="ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50" Oct 31 23:24:35.262307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.261796 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50"} err="failed to get container status \"ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50\": rpc error: code = NotFound desc = could not find container \"ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50\": container with ID starting with ef22148446fe780d455ef254b11696eb8493a1d8ac357aeb9f8bd83eb8c2cf50 not found: ID does not exist" Oct 31 23:24:35.262307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.261823 5043 scope.go:122] "RemoveContainer" containerID="b55434f93433197d088c4b7dd6b426fab9a15158df78e6eee9ce851067bddf0e" Oct 31 23:24:35.293128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.289599 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h"] Oct 31 23:24:35.293128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.292056 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-836c99-openshift-default-8f564d59f-k8d8h"] Oct 31 23:24:35.312351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.312334 5043 scope.go:122] "RemoveContainer" containerID="68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5" Oct 31 23:24:35.348766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.348747 5043 scope.go:122] "RemoveContainer" containerID="68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5" Oct 31 23:24:35.349034 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:35.349010 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5\": container with ID starting with 68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5 not found: ID does not exist" containerID="68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5" Oct 31 23:24:35.349098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.349046 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5"} err="failed to get container status \"68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5\": rpc error: code = NotFound desc = could not find container \"68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5\": container with ID starting with 68b83adc8d32609b6a7db610817b37f7e7cbf161d54e11fd0b9ec50003ab44d5 not found: ID does not exist" Oct 31 23:24:35.350377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.350355 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:24:35.443851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.443821 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.443878 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.443969 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444014 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444003 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444043 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444170 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444222 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:35.444285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444245 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444379 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444489 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket\") pod \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\" (UID: \"5f5396c0-d04b-4150-9ad9-179d7319cdb4\") " Oct 31 23:24:35.444775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444518 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:35.445004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.444979 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:35.445237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.445214 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:35.445441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.445369 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.445441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.445391 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.445441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.445406 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.445441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.445420 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.447199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.447170 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:35.447741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.447714 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "kube-api-access-jvbkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:35.448157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.448132 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:35.449082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.449055 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:35.449622 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.449595 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo" pod "5f5396c0-d04b-4150-9ad9-179d7319cdb4" (UID: "5f5396c0-d04b-4150-9ad9-179d7319cdb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:35.502441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.502400 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:35.547245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.547207 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbkz\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-kube-api-access-jvbkz\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.547245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.547241 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.547245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.547254 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.547656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.547263 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.547656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.547272 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5396c0-d04b-4150-9ad9-179d7319cdb4-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:35.637891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.637851 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:35.804250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.797120 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:35.814717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.805161 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst"] Oct 31 23:24:35.814717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.813310 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-270e8c-m2tst"] Oct 31 23:24:35.946310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.945345 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:35.957532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.949952 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" path="/var/lib/kubelet/pods/1d56bf0b-7968-4952-9917-ba0bc5cba955/volumes" Oct 31 23:24:35.957532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.950720 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4c5b40-69d4-4d04-ae24-1709b4e0896f" path="/var/lib/kubelet/pods/3c4c5b40-69d4-4d04-ae24-1709b4e0896f/volumes" Oct 31 23:24:35.957532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.951445 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" path="/var/lib/kubelet/pods/a302ec33-70dd-42ae-baa1-aaa48d20a369/volumes" Oct 31 23:24:35.957532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:35.953010 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" path="/var/lib/kubelet/pods/c2797e68-0138-44e2-bc82-71ed4daa86b5/volumes" Oct 31 23:24:36.190865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.190775 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" Oct 31 23:24:36.190865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.190827 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8" event={"ID":"5f5396c0-d04b-4150-9ad9-179d7319cdb4","Type":"ContainerDied","Data":"0acd768b08236c1bb7c451642e7dc8ca0f4a2cfd08a0beea03fbae06f9e3d46c"} Oct 31 23:24:36.191440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.190893 5043 scope.go:122] "RemoveContainer" containerID="9cc3a7985fb08998e16e68e50fad07e3c317d5287fb7b894f1ca8f689c80da4d" Oct 31 23:24:36.220559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.220525 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8"] Oct 31 23:24:36.228432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.228406 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d6685e-openshift-default-589f975f85-qdpl8"] Oct 31 23:24:36.258196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.258172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:36.775334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.775297 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4"] Oct 31 23:24:36.781962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.781931 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz"] Oct 31 23:24:36.815206 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.815167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:36.954461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:36.954425 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:37.458589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.458538 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:37.668567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.663858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:37.772809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.772769 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc"] Oct 31 23:24:37.777029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.776998 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf279c-bn2sc"] Oct 31 23:24:37.805948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.805903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:37.937039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.936998 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39da29dc-5612-402f-b061-727435a6974c" path="/var/lib/kubelet/pods/39da29dc-5612-402f-b061-727435a6974c/volumes" Oct 31 23:24:37.937709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.937686 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" path="/var/lib/kubelet/pods/5f5396c0-d04b-4150-9ad9-179d7319cdb4/volumes" Oct 31 23:24:37.944290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:37.944255 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:38.313363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:38.313328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:38.768930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:38.768896 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:39.509890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.509856 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:39.877077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.876984 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml"] Oct 31 23:24:39.882508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.882474 5043 scope.go:122] "RemoveContainer" containerID="a112283e701bccf3ade34448bb910c025ecbd7961915ee71787f35773df4709c" Oct 31 23:24:39.910820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.910638 5043 scope.go:122] "RemoveContainer" containerID="aafb50876f9cc8f75c1fdadf9b683c05e3b552873b6b0e43816a7b4e5d3e4117" Oct 31 23:24:39.932663 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.932640 5043 scope.go:122] "RemoveContainer" containerID="90e64e438bd45a3b37d8dc8a0f2035c3c8c2b25943e3e7281e357f39ed4e628c" Oct 31 23:24:39.955705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.955146 5043 scope.go:122] "RemoveContainer" containerID="6bf56397124ce158b9f46a647c9478377b71d2ee07f31193a58e705fc692976a" Oct 31 23:24:39.985379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:39.985345 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:40.126065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:40.125986 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:40.458273 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:40.458169 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:24:40.499044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:40.494238 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:40.971729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:40.971696 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:41.132568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:41.129240 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:41.301965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:41.301928 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:41.784145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:41.784096 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" containerID="cri-o://066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85" gracePeriod=25 Oct 31 23:24:41.784344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:41.784253 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" containerID="cri-o://23fca3cc4cc839371091a211e909edffc0770eba7d2c7d79b9d1d02b695df0eb" gracePeriod=25 Oct 31 23:24:41.843119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:41.843083 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:41.991807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:41.991752 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:42.148191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.144402 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:42.191650 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.191620 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g"] Oct 31 23:24:42.195390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.195368 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-7c4c91-4l86g"] Oct 31 23:24:42.333576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.332019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:42.338313 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.338280 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:24:42.341643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.341306 5043 generic.go:356] "Generic (PLEG): container finished" podID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerID="23fca3cc4cc839371091a211e909edffc0770eba7d2c7d79b9d1d02b695df0eb" exitCode=0 Oct 31 23:24:42.341643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.341508 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" event={"ID":"552eb1a2-3e7e-4311-add3-9dba9f0416af","Type":"ContainerDied","Data":"23fca3cc4cc839371091a211e909edffc0770eba7d2c7d79b9d1d02b695df0eb"} Oct 31 23:24:42.341643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.341591 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" event={"ID":"552eb1a2-3e7e-4311-add3-9dba9f0416af","Type":"ContainerDied","Data":"a745ac0d0a7eddfff1a7f43daa6a084cf1f05de7ef4775a4ba3ef54285d7dbb0"} Oct 31 23:24:42.341643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.341620 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a745ac0d0a7eddfff1a7f43daa6a084cf1f05de7ef4775a4ba3ef54285d7dbb0" Oct 31 23:24:42.344864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.344839 5043 generic.go:356] "Generic (PLEG): container finished" podID="da7b510a-9644-48e0-96fc-29447a106ee5" containerID="066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85" exitCode=0 Oct 31 23:24:42.345195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.345176 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" event={"ID":"da7b510a-9644-48e0-96fc-29447a106ee5","Type":"ContainerDied","Data":"066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85"} Oct 31 23:24:42.345343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.345330 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" event={"ID":"da7b510a-9644-48e0-96fc-29447a106ee5","Type":"ContainerDied","Data":"19cd6f8f3978f70d45cea83ec04bfa1e18676ecf7caa2a5a68e033e37834e131"} Oct 31 23:24:42.345449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.345439 5043 scope.go:122] "RemoveContainer" containerID="066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85" Oct 31 23:24:42.345727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.345708 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz" Oct 31 23:24:42.353731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.353520 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:24:42.363874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.362747 5043 scope.go:122] "RemoveContainer" containerID="066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85" Oct 31 23:24:42.363874 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:42.363112 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85\": container with ID starting with 066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85 not found: ID does not exist" containerID="066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85" Oct 31 23:24:42.363874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.363149 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85"} err="failed to get container status \"066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85\": rpc error: code = NotFound desc = could not find container \"066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85\": container with ID starting with 066105f29e3a61b6d089568dc95d26f46e103d868dcd2ca505b9237454476a85 not found: ID does not exist" Oct 31 23:24:42.455103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.455069 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s\" (UniqueName: \"kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s\") pod \"552eb1a2-3e7e-4311-add3-9dba9f0416af\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " Oct 31 23:24:42.455485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.455460 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume\") pod \"da7b510a-9644-48e0-96fc-29447a106ee5\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " Oct 31 23:24:42.456088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.456064 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume\") pod \"552eb1a2-3e7e-4311-add3-9dba9f0416af\" (UID: \"552eb1a2-3e7e-4311-add3-9dba9f0416af\") " Oct 31 23:24:42.456494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.456461 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume" pod "552eb1a2-3e7e-4311-add3-9dba9f0416af" (UID: "552eb1a2-3e7e-4311-add3-9dba9f0416af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:42.456732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.456694 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume" pod "da7b510a-9644-48e0-96fc-29447a106ee5" (UID: "da7b510a-9644-48e0-96fc-29447a106ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:42.456898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.456852 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9\" (UniqueName: \"kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9\") pod \"da7b510a-9644-48e0-96fc-29447a106ee5\" (UID: \"da7b510a-9644-48e0-96fc-29447a106ee5\") " Oct 31 23:24:42.457618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.457597 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7b510a-9644-48e0-96fc-29447a106ee5-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.457739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.457724 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552eb1a2-3e7e-4311-add3-9dba9f0416af-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.459746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.459726 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s" pod "552eb1a2-3e7e-4311-add3-9dba9f0416af" (UID: "552eb1a2-3e7e-4311-add3-9dba9f0416af"). InnerVolumeSpecName "kube-api-access-fnl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:42.460123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.460099 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9" pod "da7b510a-9644-48e0-96fc-29447a106ee5" (UID: "da7b510a-9644-48e0-96fc-29447a106ee5"). InnerVolumeSpecName "kube-api-access-66bc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:42.558679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.558607 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bc9\" (UniqueName: \"kubernetes.io/projected/da7b510a-9644-48e0-96fc-29447a106ee5-kube-api-access-66bc9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.558679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.558636 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnl9s\" (UniqueName: \"kubernetes.io/projected/552eb1a2-3e7e-4311-add3-9dba9f0416af-kube-api-access-fnl9s\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:42.677246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.677207 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz"] Oct 31 23:24:42.682837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.682809 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7c4c91-574c496d67-dznjz"] Oct 31 23:24:42.759327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:42.759293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:43.014996 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.014960 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:43.257878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.257846 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:43.356649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.356622 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4" Oct 31 23:24:43.385612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.385577 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4"] Oct 31 23:24:43.390327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.390302 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-7c4c91-587459b866-sfzf4"] Oct 31 23:24:43.399797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.399771 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:43.537747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.537657 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh"] Oct 31 23:24:43.537942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.537881 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" containerID="cri-o://bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9" gracePeriod=30 Oct 31 23:24:43.544966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.544940 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr"] Oct 31 23:24:43.545177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.545154 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" containerID="cri-o://1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d" gracePeriod=30 Oct 31 23:24:43.550870 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.550841 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx"] Oct 31 23:24:43.551074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.551050 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" containerID="cri-o://df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb" gracePeriod=30 Oct 31 23:24:43.554844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.554822 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4"] Oct 31 23:24:43.555004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.554988 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" containerID="cri-o://4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5" gracePeriod=30 Oct 31 23:24:43.593507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.593477 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht"] Oct 31 23:24:43.594214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594189 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" Oct 31 23:24:43.594214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594212 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594224 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594230 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594243 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594248 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594264 5043 cpu_manager.go:414] "containerMap: removing container" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594269 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594277 5043 cpu_manager.go:414] "containerMap: removing container" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594283 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594301 5043 cpu_manager.go:414] "containerMap: removing container" podUID="df60d0ba-d83d-47bc-b107-7b72e664e26e" containerName="revoke-keys" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594306 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df60d0ba-d83d-47bc-b107-7b72e664e26e" containerName="revoke-keys" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594318 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594323 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594332 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594337 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594346 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594350 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594361 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b53ab05c-86a3-42da-94ef-c520e1227ae8" containerName="revoke-keys" Oct 31 23:24:43.594372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594365 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53ab05c-86a3-42da-94ef-c520e1227ae8" containerName="revoke-keys" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594475 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594482 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594488 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d56bf0b-7968-4952-9917-ba0bc5cba955" containerName="payload-pre-processing" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594496 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" containerName="payload-pre-processing" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594501 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" containerName="payload-processing" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594511 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="df60d0ba-d83d-47bc-b107-7b72e664e26e" containerName="revoke-keys" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594518 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5396c0-d04b-4150-9ad9-179d7319cdb4" containerName="istio-proxy" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594524 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2797e68-0138-44e2-bc82-71ed4daa86b5" containerName="istio-proxy" Oct 31 23:24:43.595007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.594531 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53ab05c-86a3-42da-94ef-c520e1227ae8" containerName="revoke-keys" Oct 31 23:24:43.599516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.599485 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn"] Oct 31 23:24:43.599693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.599668 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.600748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.600724 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a302ec33-70dd-42ae-baa1-aaa48d20a369" containerName="istio-proxy" Oct 31 23:24:43.616946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.616912 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:43.617111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.616964 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:24:43.617111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.617021 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:43.617111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.617080 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:24:43.623649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.623627 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht"] Oct 31 23:24:43.623857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.623844 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.634791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.634767 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn"] Oct 31 23:24:43.648651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.643232 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:43.692871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.691788 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.692871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.691835 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqsj\" (UniqueName: \"kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.692871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.691906 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.692871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.692000 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfzd\" (UniqueName: \"kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.692871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.692090 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.692871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.692129 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.793539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.793432 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.793539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.793518 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.793770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.793695 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.793770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.793731 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqsj\" (UniqueName: \"kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.793849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.793796 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.793908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.793892 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfzd\" (UniqueName: \"kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.794830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.794807 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.794931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.794831 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.797075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.797050 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.797710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.797687 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.806979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.806957 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfzd\" (UniqueName: \"kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd\") pod \"maas-api-revoke-sub-207ce6f3-zp9dn\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:43.807212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.807190 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqsj\" (UniqueName: \"kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj\") pod \"maas-api-revoke-sub-a2980884-7kdht\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.934854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.934824 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:43.941581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.941516 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552eb1a2-3e7e-4311-add3-9dba9f0416af" path="/var/lib/kubelet/pods/552eb1a2-3e7e-4311-add3-9dba9f0416af/volumes" Oct 31 23:24:43.942180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.942154 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7b510a-9644-48e0-96fc-29447a106ee5" path="/var/lib/kubelet/pods/da7b510a-9644-48e0-96fc-29447a106ee5/volumes" Oct 31 23:24:43.942786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:43.942765 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85c9373-7a8b-4cb7-919a-d0f9f451b383" path="/var/lib/kubelet/pods/f85c9373-7a8b-4cb7-919a-d0f9f451b383/volumes" Oct 31 23:24:44.027327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.002897 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:44.099383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.099341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:44.158610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.153958 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6"] Oct 31 23:24:44.178999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.178954 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6"] Oct 31 23:24:44.179140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.179113 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.205842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.205491 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.206184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.206157 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gr7x\" (UniqueName: \"kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.206334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.206312 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.254828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.254569 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:44.264422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.264393 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht"] Oct 31 23:24:44.312431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.312400 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gr7x\" (UniqueName: \"kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.312599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.312454 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.313243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.313219 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.313452 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.313414 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.317906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.317877 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.367420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.367382 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" event={"ID":"1d723e9b-2145-4fb4-a430-e7c6982b1f5b","Type":"ContainerStarted","Data":"1a5a8861cdc0e70ea3f4cb4c3aa249e221200ab3d74f834ee21a7492dacca5d1"} Oct 31 23:24:44.414885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.414853 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gr7x\" (UniqueName: \"kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x\") pod \"maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.441344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.441317 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn"] Oct 31 23:24:44.441596 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:44.441569 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4272e612_4718_4d0b_99b1_da0907a3865d.slice/crio-6ae5e46896f1fdc95e49d14a35ced21e48ddacc32300abc28b8bdb5eced1f2a2 WatchSource:0}: Error finding container 6ae5e46896f1fdc95e49d14a35ced21e48ddacc32300abc28b8bdb5eced1f2a2: Status 404 returned error can't find the container with id 6ae5e46896f1fdc95e49d14a35ced21e48ddacc32300abc28b8bdb5eced1f2a2 Oct 31 23:24:44.511541 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.511513 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:44.623920 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.623893 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb"] Oct 31 23:24:44.638516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.634416 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.653440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.651988 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb"] Oct 31 23:24:44.725485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.725449 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.725722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.725699 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.725841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.725822 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrzt\" (UniqueName: \"kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.827081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.827049 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.827280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.827163 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrzt\" (UniqueName: \"kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.827280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.827256 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.827902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.827879 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.829728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.829700 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.843090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.843057 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrzt\" (UniqueName: \"kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt\") pod \"maas-api-revoke-sub-ab3c0a48-5fxsb\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:44.877740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.877667 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" containerID="cri-o://a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe" gracePeriod=25 Oct 31 23:24:44.919918 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:44.919876 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1de0a86_4a0b_4429_9088_6f6bf329b42c.slice/crio-643e350bb7d230e65a7654d1e967a96c4df8d81967f68d180b0ac373350de5e3 WatchSource:0}: Error finding container 643e350bb7d230e65a7654d1e967a96c4df8d81967f68d180b0ac373350de5e3: Status 404 returned error can't find the container with id 643e350bb7d230e65a7654d1e967a96c4df8d81967f68d180b0ac373350de5e3 Oct 31 23:24:44.921776 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.921752 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6"] Oct 31 23:24:44.963596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.957729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:44.963596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:44.963481 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:45.290381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.290333 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:45.303366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.302697 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb"] Oct 31 23:24:45.379120 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.379088 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" event={"ID":"4272e612-4718-4d0b-99b1-da0907a3865d","Type":"ContainerStarted","Data":"0c32d3b60fc21517265a73f7e08fb2b1e613677fa037cca5852f44e8be53d1da"} Oct 31 23:24:45.379256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.379148 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" event={"ID":"4272e612-4718-4d0b-99b1-da0907a3865d","Type":"ContainerStarted","Data":"6ae5e46896f1fdc95e49d14a35ced21e48ddacc32300abc28b8bdb5eced1f2a2"} Oct 31 23:24:45.380523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.380493 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" event={"ID":"d1de0a86-4a0b-4429-9088-6f6bf329b42c","Type":"ContainerStarted","Data":"643e350bb7d230e65a7654d1e967a96c4df8d81967f68d180b0ac373350de5e3"} Oct 31 23:24:45.382417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.382390 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" event={"ID":"1d723e9b-2145-4fb4-a430-e7c6982b1f5b","Type":"ContainerStarted","Data":"f5e014abf7b6edaba9a0788a545a9bab5504063dab06b598844184617e9822b7"} Oct 31 23:24:45.383954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.383928 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" event={"ID":"f74e6297-4386-49eb-bfc6-a86142b3fc2e","Type":"ContainerStarted","Data":"ae854fe847144b4f8c233d8854907ec2634352f0440ddce1b29bf8675f96de8f"} Oct 31 23:24:45.403141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.403033 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" podStartSLOduration=2.403012189 podStartE2EDuration="2.403012189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:45.398984481 +0000 UTC m=+2111.628538891" watchObservedRunningTime="2026-10-31 23:24:45.403012189 +0000 UTC m=+2111.632566662" Oct 31 23:24:45.438850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.436263 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" podStartSLOduration=2.436242148 podStartE2EDuration="2.436242148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:45.426794817 +0000 UTC m=+2111.656349230" watchObservedRunningTime="2026-10-31 23:24:45.436242148 +0000 UTC m=+2111.665796563" Oct 31 23:24:45.460488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.460449 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:45.583806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.583764 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:24:45.613042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.612984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:45.762734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:45.761259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:46.048692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.048661 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2"] Oct 31 23:24:46.173830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.143275 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2"] Oct 31 23:24:46.173830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.143702 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.187770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.180248 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27sln\" (UniqueName: \"kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.217569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.200032 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.217569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.200125 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.302099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.302060 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.302639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.302142 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.302639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.302239 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27sln\" (UniqueName: \"kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.303094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.303074 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.304707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.304684 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.315453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.315422 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27sln\" (UniqueName: \"kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.396796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.396757 5043 generic.go:356] "Generic (PLEG): container finished" podID="d1de0a86-4a0b-4429-9088-6f6bf329b42c" containerID="875e3b5b0ef3027b41663f0bb0f0d549fb1db638b3002904340e0f452074d13b" exitCode=0 Oct 31 23:24:46.396970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.396840 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" event={"ID":"d1de0a86-4a0b-4429-9088-6f6bf329b42c","Type":"ContainerDied","Data":"875e3b5b0ef3027b41663f0bb0f0d549fb1db638b3002904340e0f452074d13b"} Oct 31 23:24:46.398681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.398643 5043 generic.go:356] "Generic (PLEG): container finished" podID="1d723e9b-2145-4fb4-a430-e7c6982b1f5b" containerID="f5e014abf7b6edaba9a0788a545a9bab5504063dab06b598844184617e9822b7" exitCode=0 Oct 31 23:24:46.398801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.398722 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" event={"ID":"1d723e9b-2145-4fb4-a430-e7c6982b1f5b","Type":"ContainerDied","Data":"f5e014abf7b6edaba9a0788a545a9bab5504063dab06b598844184617e9822b7"} Oct 31 23:24:46.400404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.400381 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" event={"ID":"f74e6297-4386-49eb-bfc6-a86142b3fc2e","Type":"ContainerStarted","Data":"579cfc08fc16c3145b4a14a28ec9c3544567455d156615b2fb9c02e239d51f63"} Oct 31 23:24:46.402180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.402151 5043 generic.go:356] "Generic (PLEG): container finished" podID="4272e612-4718-4d0b-99b1-da0907a3865d" containerID="0c32d3b60fc21517265a73f7e08fb2b1e613677fa037cca5852f44e8be53d1da" exitCode=0 Oct 31 23:24:46.402275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.402173 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" event={"ID":"4272e612-4718-4d0b-99b1-da0907a3865d","Type":"ContainerDied","Data":"0c32d3b60fc21517265a73f7e08fb2b1e613677fa037cca5852f44e8be53d1da"} Oct 31 23:24:46.459374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.459325 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" podStartSLOduration=2.459310799 podStartE2EDuration="2.459310799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:46.455439368 +0000 UTC m=+2112.684993780" watchObservedRunningTime="2026-10-31 23:24:46.459310799 +0000 UTC m=+2112.688865236" Oct 31 23:24:46.566444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.566409 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:46.790911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.790759 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:46.883917 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:46.883879 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc995810_97ff_47a0_aea1_1a4b76f4b7d4.slice/crio-4f54cc939b7d704dc5916840f99879318836ea6a3a5238ce4bbbaba35b6aa8c9 WatchSource:0}: Error finding container 4f54cc939b7d704dc5916840f99879318836ea6a3a5238ce4bbbaba35b6aa8c9: Status 404 returned error can't find the container with id 4f54cc939b7d704dc5916840f99879318836ea6a3a5238ce4bbbaba35b6aa8c9 Oct 31 23:24:46.884352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:46.884324 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2"] Oct 31 23:24:47.412531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.412435 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" event={"ID":"cc995810-97ff-47a0-aea1-1a4b76f4b7d4","Type":"ContainerStarted","Data":"1a8337a748a5c171f61a37f35413ba1fe5f6449ce1923dcc072d4d56835262f2"} Oct 31 23:24:47.412531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.412498 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" event={"ID":"cc995810-97ff-47a0-aea1-1a4b76f4b7d4","Type":"ContainerStarted","Data":"4f54cc939b7d704dc5916840f99879318836ea6a3a5238ce4bbbaba35b6aa8c9"} Oct 31 23:24:47.414395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.414343 5043 generic.go:356] "Generic (PLEG): container finished" podID="f74e6297-4386-49eb-bfc6-a86142b3fc2e" containerID="579cfc08fc16c3145b4a14a28ec9c3544567455d156615b2fb9c02e239d51f63" exitCode=0 Oct 31 23:24:47.414521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.414428 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" event={"ID":"f74e6297-4386-49eb-bfc6-a86142b3fc2e","Type":"ContainerDied","Data":"579cfc08fc16c3145b4a14a28ec9c3544567455d156615b2fb9c02e239d51f63"} Oct 31 23:24:47.433196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.433143 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" podStartSLOduration=1.433130176 podStartE2EDuration="1.433130176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:47.430626389 +0000 UTC m=+2113.660180802" watchObservedRunningTime="2026-10-31 23:24:47.433130176 +0000 UTC m=+2113.662684634" Oct 31 23:24:47.525688 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.511858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:47.662999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.662935 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:47.730665 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.730634 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:47.743984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.743956 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj\" (UniqueName: \"kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj\") pod \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " Oct 31 23:24:47.744162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.744112 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca\") pod \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " Oct 31 23:24:47.744228 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.744216 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca\") pod \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\" (UID: \"1d723e9b-2145-4fb4-a430-e7c6982b1f5b\") " Oct 31 23:24:47.744701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.744670 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca" pod "1d723e9b-2145-4fb4-a430-e7c6982b1f5b" (UID: "1d723e9b-2145-4fb4-a430-e7c6982b1f5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:47.744981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.744946 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.746331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.746307 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj" pod "1d723e9b-2145-4fb4-a430-e7c6982b1f5b" (UID: "1d723e9b-2145-4fb4-a430-e7c6982b1f5b"). InnerVolumeSpecName "kube-api-access-tsqsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:47.746402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.746377 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca" pod "1d723e9b-2145-4fb4-a430-e7c6982b1f5b" (UID: "1d723e9b-2145-4fb4-a430-e7c6982b1f5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:47.780653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.780625 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:47.845700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.845670 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd\" (UniqueName: \"kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd\") pod \"4272e612-4718-4d0b-99b1-da0907a3865d\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " Oct 31 23:24:47.845866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.845777 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca\") pod \"4272e612-4718-4d0b-99b1-da0907a3865d\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " Oct 31 23:24:47.845907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.845873 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca\") pod \"4272e612-4718-4d0b-99b1-da0907a3865d\" (UID: \"4272e612-4718-4d0b-99b1-da0907a3865d\") " Oct 31 23:24:47.846283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.846252 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca" pod "4272e612-4718-4d0b-99b1-da0907a3865d" (UID: "4272e612-4718-4d0b-99b1-da0907a3865d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:47.846509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.846489 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqsj\" (UniqueName: \"kubernetes.io/projected/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-kube-api-access-tsqsj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.846595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.846512 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.846595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.846531 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d723e9b-2145-4fb4-a430-e7c6982b1f5b-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.914219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.914035 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca" pod "4272e612-4718-4d0b-99b1-da0907a3865d" (UID: "4272e612-4718-4d0b-99b1-da0907a3865d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:47.914219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.914092 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd" pod "4272e612-4718-4d0b-99b1-da0907a3865d" (UID: "4272e612-4718-4d0b-99b1-da0907a3865d"). InnerVolumeSpecName "kube-api-access-2lfzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:47.947501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.947466 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca\") pod \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " Oct 31 23:24:47.947711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.947601 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x\" (UniqueName: \"kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x\") pod \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " Oct 31 23:24:47.947711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.947691 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca\") pod \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\" (UID: \"d1de0a86-4a0b-4429-9088-6f6bf329b42c\") " Oct 31 23:24:47.947892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.947857 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca" pod "d1de0a86-4a0b-4429-9088-6f6bf329b42c" (UID: "d1de0a86-4a0b-4429-9088-6f6bf329b42c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:47.948335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.948314 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfzd\" (UniqueName: \"kubernetes.io/projected/4272e612-4718-4d0b-99b1-da0907a3865d-kube-api-access-2lfzd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.948425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.948341 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4272e612-4718-4d0b-99b1-da0907a3865d-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.948425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.948361 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:47.949793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.949765 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca" pod "d1de0a86-4a0b-4429-9088-6f6bf329b42c" (UID: "d1de0a86-4a0b-4429-9088-6f6bf329b42c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:47.949921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:47.949855 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x" pod "d1de0a86-4a0b-4429-9088-6f6bf329b42c" (UID: "d1de0a86-4a0b-4429-9088-6f6bf329b42c"). InnerVolumeSpecName "kube-api-access-8gr7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:48.049948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.049914 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1de0a86-4a0b-4429-9088-6f6bf329b42c-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:48.050068 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.049951 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gr7x\" (UniqueName: \"kubernetes.io/projected/d1de0a86-4a0b-4429-9088-6f6bf329b42c-kube-api-access-8gr7x\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:48.128568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.128519 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:48.426392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.426360 5043 generic.go:356] "Generic (PLEG): container finished" podID="cc995810-97ff-47a0-aea1-1a4b76f4b7d4" containerID="1a8337a748a5c171f61a37f35413ba1fe5f6449ce1923dcc072d4d56835262f2" exitCode=0 Oct 31 23:24:48.426839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.426445 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" event={"ID":"cc995810-97ff-47a0-aea1-1a4b76f4b7d4","Type":"ContainerDied","Data":"1a8337a748a5c171f61a37f35413ba1fe5f6449ce1923dcc072d4d56835262f2"} Oct 31 23:24:48.428146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.428119 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" Oct 31 23:24:48.428299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.428158 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht" event={"ID":"1d723e9b-2145-4fb4-a430-e7c6982b1f5b","Type":"ContainerDied","Data":"1a5a8861cdc0e70ea3f4cb4c3aa249e221200ab3d74f834ee21a7492dacca5d1"} Oct 31 23:24:48.428299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.428202 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a5a8861cdc0e70ea3f4cb4c3aa249e221200ab3d74f834ee21a7492dacca5d1" Oct 31 23:24:48.429860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.429841 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" Oct 31 23:24:48.430013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.429902 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn" event={"ID":"4272e612-4718-4d0b-99b1-da0907a3865d","Type":"ContainerDied","Data":"6ae5e46896f1fdc95e49d14a35ced21e48ddacc32300abc28b8bdb5eced1f2a2"} Oct 31 23:24:48.430013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.429941 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae5e46896f1fdc95e49d14a35ced21e48ddacc32300abc28b8bdb5eced1f2a2" Oct 31 23:24:48.431796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.431779 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" Oct 31 23:24:48.432007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.431802 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6" event={"ID":"d1de0a86-4a0b-4429-9088-6f6bf329b42c","Type":"ContainerDied","Data":"643e350bb7d230e65a7654d1e967a96c4df8d81967f68d180b0ac373350de5e3"} Oct 31 23:24:48.432100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.432033 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643e350bb7d230e65a7654d1e967a96c4df8d81967f68d180b0ac373350de5e3" Oct 31 23:24:48.841842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.839508 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:48.846368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.846251 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891253 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891312 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891363 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891411 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891491 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891528 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891602 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt\" (UniqueName: \"kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt\") pod \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891648 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca\") pod \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891817 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.891854 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.892851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.892301 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca\") pod \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\" (UID: \"f74e6297-4386-49eb-bfc6-a86142b3fc2e\") " Oct 31 23:24:48.892851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.892369 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert\") pod \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\" (UID: \"4ba4a82c-f1bf-4f69-9763-007d39edc7cd\") " Oct 31 23:24:48.893589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.893490 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:48.896384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.896353 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:48.900806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.899557 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca" pod "f74e6297-4386-49eb-bfc6-a86142b3fc2e" (UID: "f74e6297-4386-49eb-bfc6-a86142b3fc2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:48.900806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.899781 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:48.900806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.899935 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.900806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.900468 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.900806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.900714 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.900806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.900781 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:48.904018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.903991 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca" pod "f74e6297-4386-49eb-bfc6-a86142b3fc2e" (UID: "f74e6297-4386-49eb-bfc6-a86142b3fc2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:48.906186 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.906150 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt" pod "f74e6297-4386-49eb-bfc6-a86142b3fc2e" (UID: "f74e6297-4386-49eb-bfc6-a86142b3fc2e"). InnerVolumeSpecName "kube-api-access-flrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:48.958358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.931044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:48.958358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.933404 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "kube-api-access-lck6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:48.958358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.936690 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:24:48.977276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.976967 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994412 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994475 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994503 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994563 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994738 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994784 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994850 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.994987 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995069 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995148 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995222 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995279 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995335 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995401 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.995480 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.996099 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.996141 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs\") pod \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\" (UID: \"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.996185 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token\") pod \"e1034845-c776-4a60-acb5-b94609230c97\" (UID: \"e1034845-c776-4a60-acb5-b94609230c97\") " Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997048 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997072 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997091 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997105 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997123 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997139 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997156 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997170 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lck6w\" (UniqueName: \"kubernetes.io/projected/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-kube-api-access-lck6w\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997185 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997199 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrzt\" (UniqueName: \"kubernetes.io/projected/f74e6297-4386-49eb-bfc6-a86142b3fc2e-kube-api-access-flrzt\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.000804 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:48.997212 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74e6297-4386-49eb-bfc6-a86142b3fc2e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.000942 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.000957 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.001167 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.001355 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.002207 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.003070 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.005882 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.006126 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.006373 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:49.006535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.006462 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.018466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.015766 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8"] Oct 31 23:24:49.029192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.029153 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.029729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.029694 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.030393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.029941 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.030393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.030358 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "kube-api-access-w6ql6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.030837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.030814 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z"] Oct 31 23:24:49.030940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.030811 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy" pod "4ba4a82c-f1bf-4f69-9763-007d39edc7cd" (UID: "4ba4a82c-f1bf-4f69-9763-007d39edc7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.031338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.031311 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "kube-api-access-lb7h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.032786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.032759 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.032882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.032814 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo" pod "e1034845-c776-4a60-acb5-b94609230c97" (UID: "e1034845-c776-4a60-acb5-b94609230c97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:49.033167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.033140 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo" pod "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" (UID: "c6d168ac-9f41-4bec-9146-9ac41e9ad9d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:49.092495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.092467 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:24:49.098164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098136 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6ql6\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-kube-api-access-w6ql6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098166 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098178 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098187 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098196 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098205 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a82c-f1bf-4f69-9763-007d39edc7cd-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098213 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098221 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098234 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034845-c776-4a60-acb5-b94609230c97-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098248 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034845-c776-4a60-acb5-b94609230c97-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098257 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098266 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098274 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098282 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098290 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034845-c776-4a60-acb5-b94609230c97-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098298 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7h5\" (UniqueName: \"kubernetes.io/projected/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-kube-api-access-lb7h5\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098308 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098321 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.098353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.098331 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034845-c776-4a60-acb5-b94609230c97-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.137496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.137451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:49.199498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199469 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199517 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199578 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199642 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199676 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199774 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199794 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.199966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199808 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.199966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199844 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.200200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199973 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.200200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.199982 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:49.200200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200086 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.200368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200202 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs\") pod \"09b6bd84-4b59-4823-8613-8b6519a6db0d\" (UID: \"09b6bd84-4b59-4823-8613-8b6519a6db0d\") " Oct 31 23:24:49.200505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200483 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.200954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200926 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.200954 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200955 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.201135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200969 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b6bd84-4b59-4823-8613-8b6519a6db0d-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.201135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200984 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.201135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.200998 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.202194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.202171 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:24:49.202382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.202362 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "kube-api-access-k99f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.202457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.202402 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:49.202457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.202403 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token" pod "09b6bd84-4b59-4823-8613-8b6519a6db0d" (UID: "09b6bd84-4b59-4823-8613-8b6519a6db0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.302023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.301985 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99f2\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-kube-api-access-k99f2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.302023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.302021 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.302233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.302034 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.302233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.302042 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b6bd84-4b59-4823-8613-8b6519a6db0d-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.441600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.441559 5043 generic.go:356] "Generic (PLEG): container finished" podID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerID="bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9" exitCode=0 Oct 31 23:24:49.442078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.441633 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" Oct 31 23:24:49.442078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.441655 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" event={"ID":"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4","Type":"ContainerDied","Data":"bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9"} Oct 31 23:24:49.442078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.441744 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh" event={"ID":"c6d168ac-9f41-4bec-9146-9ac41e9ad9d4","Type":"ContainerDied","Data":"3eea2833f04aac4c1fe6d079503e87706000894ef93ef8ca27d3ddc04811768c"} Oct 31 23:24:49.442078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.441772 5043 scope.go:122] "RemoveContainer" containerID="bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9" Oct 31 23:24:49.443666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.443641 5043 generic.go:356] "Generic (PLEG): container finished" podID="e1034845-c776-4a60-acb5-b94609230c97" containerID="df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb" exitCode=0 Oct 31 23:24:49.443765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.443718 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" Oct 31 23:24:49.443765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.443750 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" event={"ID":"e1034845-c776-4a60-acb5-b94609230c97","Type":"ContainerDied","Data":"df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb"} Oct 31 23:24:49.443872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.443801 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx" event={"ID":"e1034845-c776-4a60-acb5-b94609230c97","Type":"ContainerDied","Data":"eead1cb941e2cdb366ba9ca1d66b8274c6c91094fc7ed162dab2cbe702c93c2b"} Oct 31 23:24:49.445428 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.445404 5043 generic.go:356] "Generic (PLEG): container finished" podID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerID="1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d" exitCode=0 Oct 31 23:24:49.445590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.445483 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" event={"ID":"4ba4a82c-f1bf-4f69-9763-007d39edc7cd","Type":"ContainerDied","Data":"1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d"} Oct 31 23:24:49.445590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.445522 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" event={"ID":"4ba4a82c-f1bf-4f69-9763-007d39edc7cd","Type":"ContainerDied","Data":"a9ab8b4e4e7ba721a3490e72a08c30ce88fd370d03f36aea34dad29a9bfc718c"} Oct 31 23:24:49.445712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.445615 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr" Oct 31 23:24:49.450612 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.450593 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" Oct 31 23:24:49.450890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.450823 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb" event={"ID":"f74e6297-4386-49eb-bfc6-a86142b3fc2e","Type":"ContainerDied","Data":"ae854fe847144b4f8c233d8854907ec2634352f0440ddce1b29bf8675f96de8f"} Oct 31 23:24:49.450890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.450869 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae854fe847144b4f8c233d8854907ec2634352f0440ddce1b29bf8675f96de8f" Oct 31 23:24:49.452439 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.452337 5043 generic.go:356] "Generic (PLEG): container finished" podID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerID="4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5" exitCode=0 Oct 31 23:24:49.452737 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.452713 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" Oct 31 23:24:49.454529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.453073 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" event={"ID":"09b6bd84-4b59-4823-8613-8b6519a6db0d","Type":"ContainerDied","Data":"4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5"} Oct 31 23:24:49.454529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.453130 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4" event={"ID":"09b6bd84-4b59-4823-8613-8b6519a6db0d","Type":"ContainerDied","Data":"12d28136467e1b184b5a59ff55c949480777fcd0ece2249e6888a69b1ad82349"} Oct 31 23:24:49.458248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.458227 5043 scope.go:122] "RemoveContainer" containerID="bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9" Oct 31 23:24:49.459325 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:49.459303 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9\": container with ID starting with bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9 not found: ID does not exist" containerID="bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9" Oct 31 23:24:49.459426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.459335 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9"} err="failed to get container status \"bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9\": rpc error: code = NotFound desc = could not find container \"bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9\": container with ID starting with bcdca4c567c70e2d12e83f5362a7fd13110f796d87ad0414159fbda21d1cadf9 not found: ID does not exist" Oct 31 23:24:49.459426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.459356 5043 scope.go:122] "RemoveContainer" containerID="df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb" Oct 31 23:24:49.474013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.473994 5043 scope.go:122] "RemoveContainer" containerID="df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb" Oct 31 23:24:49.474296 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:49.474279 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb\": container with ID starting with df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb not found: ID does not exist" containerID="df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb" Oct 31 23:24:49.474349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.474303 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb"} err="failed to get container status \"df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb\": rpc error: code = NotFound desc = could not find container \"df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb\": container with ID starting with df50addd236e031e5e290ad978aea5ddb20be6bb1707370db51378b5c1b4b3eb not found: ID does not exist" Oct 31 23:24:49.474349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.474319 5043 scope.go:122] "RemoveContainer" containerID="1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d" Oct 31 23:24:49.483241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.483220 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh"] Oct 31 23:24:49.484885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.484864 5043 scope.go:122] "RemoveContainer" containerID="1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d" Oct 31 23:24:49.485326 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:49.485299 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d\": container with ID starting with 1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d not found: ID does not exist" containerID="1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d" Oct 31 23:24:49.485392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.485337 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d"} err="failed to get container status \"1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d\": rpc error: code = NotFound desc = could not find container \"1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d\": container with ID starting with 1c30ab7b1aaf62853ba446a2dcbeeed597e3f2e8ff5fd669c8098e4dfae99c8d not found: ID does not exist" Oct 31 23:24:49.485392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.485361 5043 scope.go:122] "RemoveContainer" containerID="4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5" Oct 31 23:24:49.494582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.494559 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-96vkh"] Oct 31 23:24:49.498090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.498068 5043 scope.go:122] "RemoveContainer" containerID="4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5" Oct 31 23:24:49.498371 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:49.498351 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5\": container with ID starting with 4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5 not found: ID does not exist" containerID="4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5" Oct 31 23:24:49.498432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.498377 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5"} err="failed to get container status \"4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5\": rpc error: code = NotFound desc = could not find container \"4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5\": container with ID starting with 4edcef063bfc28f8952867e1e61f4353259d29cfd79484115c7e00f63bebf3e5 not found: ID does not exist" Oct 31 23:24:49.512309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.512287 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4"] Oct 31 23:24:49.516922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.516899 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-45xt4"] Oct 31 23:24:49.532614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.532588 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr"] Oct 31 23:24:49.536980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.536959 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x2btr"] Oct 31 23:24:49.563674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.563609 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx"] Oct 31 23:24:49.565949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.565923 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-270e8c-openshift-default-77f44cd8b5-x82jx"] Oct 31 23:24:49.613891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.613864 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:49.642984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.642936 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" containerID="cri-o://264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9" gracePeriod=45 Oct 31 23:24:49.718512 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.718393 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca\") pod \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " Oct 31 23:24:49.719216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.718460 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca\") pod \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " Oct 31 23:24:49.719216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.718932 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln\" (UniqueName: \"kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln\") pod \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\" (UID: \"cc995810-97ff-47a0-aea1-1a4b76f4b7d4\") " Oct 31 23:24:49.723272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.723240 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca" pod "cc995810-97ff-47a0-aea1-1a4b76f4b7d4" (UID: "cc995810-97ff-47a0-aea1-1a4b76f4b7d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:49.728935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.728902 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:49.732893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.732866 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln" pod "cc995810-97ff-47a0-aea1-1a4b76f4b7d4" (UID: "cc995810-97ff-47a0-aea1-1a4b76f4b7d4"). InnerVolumeSpecName "kube-api-access-27sln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:49.734613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.734516 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca" pod "cc995810-97ff-47a0-aea1-1a4b76f4b7d4" (UID: "cc995810-97ff-47a0-aea1-1a4b76f4b7d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:49.824161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.824061 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27sln\" (UniqueName: \"kubernetes.io/projected/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-kube-api-access-27sln\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.824161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.824104 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.824161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.824120 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc995810-97ff-47a0-aea1-1a4b76f4b7d4-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:49.907853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.907827 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:24:49.942628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.942590 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" path="/var/lib/kubelet/pods/09b6bd84-4b59-4823-8613-8b6519a6db0d/volumes" Oct 31 23:24:49.943341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.943320 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" path="/var/lib/kubelet/pods/4ba4a82c-f1bf-4f69-9763-007d39edc7cd/volumes" Oct 31 23:24:49.943993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.943970 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" path="/var/lib/kubelet/pods/c6d168ac-9f41-4bec-9146-9ac41e9ad9d4/volumes" Oct 31 23:24:49.945321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:49.945299 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1034845-c776-4a60-acb5-b94609230c97" path="/var/lib/kubelet/pods/e1034845-c776-4a60-acb5-b94609230c97/volumes" Oct 31 23:24:50.026767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.026715 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache\") pod \"160baa62-7a99-44ef-ac94-58f851fbc7a5\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " Oct 31 23:24:50.026957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.026799 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm\") pod \"160baa62-7a99-44ef-ac94-58f851fbc7a5\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " Oct 31 23:24:50.026957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.026933 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs\") pod \"160baa62-7a99-44ef-ac94-58f851fbc7a5\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " Oct 31 23:24:50.027079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.026963 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir\") pod \"160baa62-7a99-44ef-ac94-58f851fbc7a5\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " Oct 31 23:24:50.027079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.027010 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home\") pod \"160baa62-7a99-44ef-ac94-58f851fbc7a5\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " Oct 31 23:24:50.027079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.027040 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx\") pod \"160baa62-7a99-44ef-ac94-58f851fbc7a5\" (UID: \"160baa62-7a99-44ef-ac94-58f851fbc7a5\") " Oct 31 23:24:50.029028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.028945 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home" pod "160baa62-7a99-44ef-ac94-58f851fbc7a5" (UID: "160baa62-7a99-44ef-ac94-58f851fbc7a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.029184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.029158 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir" pod "160baa62-7a99-44ef-ac94-58f851fbc7a5" (UID: "160baa62-7a99-44ef-ac94-58f851fbc7a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.029333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.029225 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache" pod "160baa62-7a99-44ef-ac94-58f851fbc7a5" (UID: "160baa62-7a99-44ef-ac94-58f851fbc7a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.034681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.034647 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx" pod "160baa62-7a99-44ef-ac94-58f851fbc7a5" (UID: "160baa62-7a99-44ef-ac94-58f851fbc7a5"). InnerVolumeSpecName "kube-api-access-vjxdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:50.039402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.039375 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs" pod "160baa62-7a99-44ef-ac94-58f851fbc7a5" (UID: "160baa62-7a99-44ef-ac94-58f851fbc7a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:50.042198 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.042161 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm" pod "160baa62-7a99-44ef-ac94-58f851fbc7a5" (UID: "160baa62-7a99-44ef-ac94-58f851fbc7a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:24:50.128729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.128648 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160baa62-7a99-44ef-ac94-58f851fbc7a5-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.128729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.128687 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.128729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.128701 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.128729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.128715 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxdx\" (UniqueName: \"kubernetes.io/projected/160baa62-7a99-44ef-ac94-58f851fbc7a5-kube-api-access-vjxdx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.128729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.128731 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.128997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.128745 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160baa62-7a99-44ef-ac94-58f851fbc7a5-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:50.292947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.292903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:50.468684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.468580 5043 generic.go:356] "Generic (PLEG): container finished" podID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerID="264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9" exitCode=0 Oct 31 23:24:50.468684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.468640 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" Oct 31 23:24:50.468684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.468671 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" event={"ID":"160baa62-7a99-44ef-ac94-58f851fbc7a5","Type":"ContainerDied","Data":"264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9"} Oct 31 23:24:50.469275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.468716 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv" event={"ID":"160baa62-7a99-44ef-ac94-58f851fbc7a5","Type":"ContainerDied","Data":"44004b0a404a7c8cd1cf722fcaf0578bf4c2b385dae2d0a8647172e95eb13a27"} Oct 31 23:24:50.469275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.468737 5043 scope.go:122] "RemoveContainer" containerID="264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9" Oct 31 23:24:50.472631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.472301 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" event={"ID":"cc995810-97ff-47a0-aea1-1a4b76f4b7d4","Type":"ContainerDied","Data":"4f54cc939b7d704dc5916840f99879318836ea6a3a5238ce4bbbaba35b6aa8c9"} Oct 31 23:24:50.472631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.472354 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f54cc939b7d704dc5916840f99879318836ea6a3a5238ce4bbbaba35b6aa8c9" Oct 31 23:24:50.472631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.472313 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2" Oct 31 23:24:50.490711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.490689 5043 scope.go:122] "RemoveContainer" containerID="264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9" Oct 31 23:24:50.491053 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:24:50.491031 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9\": container with ID starting with 264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9 not found: ID does not exist" containerID="264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9" Oct 31 23:24:50.491158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.491061 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9"} err="failed to get container status \"264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9\": rpc error: code = NotFound desc = could not find container \"264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9\": container with ID starting with 264b3aea71e5075506633c6cfe2e8ee703a181078ed601750f922193d37f51b9 not found: ID does not exist" Oct 31 23:24:50.559844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.559811 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv"] Oct 31 23:24:50.585685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.585637 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:50.586848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.586819 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7d7eed/premium-w6-7d7eed-kserve-5599dcf654-vx4rv"] Oct 31 23:24:50.732966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:50.732833 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:51.230164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.230126 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:51.339311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.338928 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp"] Oct 31 23:24:51.344070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344036 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" Oct 31 23:24:51.344070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344069 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344093 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d1de0a86-4a0b-4429-9088-6f6bf329b42c" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344102 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1de0a86-4a0b-4429-9088-6f6bf329b42c" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344129 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4272e612-4718-4d0b-99b1-da0907a3865d" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344138 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4272e612-4718-4d0b-99b1-da0907a3865d" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344155 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1d723e9b-2145-4fb4-a430-e7c6982b1f5b" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344163 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d723e9b-2145-4fb4-a430-e7c6982b1f5b" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344177 5043 cpu_manager.go:414] "containerMap: removing container" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344184 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344204 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344211 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344234 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cc995810-97ff-47a0-aea1-1a4b76f4b7d4" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344242 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc995810-97ff-47a0-aea1-1a4b76f4b7d4" containerName="revoke-keys" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344290 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344299 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344311 5043 cpu_manager.go:414] "containerMap: removing container" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344318 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" Oct 31 23:24:51.344321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344328 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f74e6297-4386-49eb-bfc6-a86142b3fc2e" containerName="revoke-keys" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344336 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74e6297-4386-49eb-bfc6-a86142b3fc2e" containerName="revoke-keys" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344567 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4272e612-4718-4d0b-99b1-da0907a3865d" containerName="revoke-keys" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344589 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d723e9b-2145-4fb4-a430-e7c6982b1f5b" containerName="revoke-keys" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344601 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b6bd84-4b59-4823-8613-8b6519a6db0d" containerName="istio-proxy" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344613 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1034845-c776-4a60-acb5-b94609230c97" containerName="istio-proxy" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344626 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc995810-97ff-47a0-aea1-1a4b76f4b7d4" containerName="revoke-keys" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344641 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba4a82c-f1bf-4f69-9763-007d39edc7cd" containerName="istio-proxy" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344651 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" containerName="main" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344660 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74e6297-4386-49eb-bfc6-a86142b3fc2e" containerName="revoke-keys" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344674 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d168ac-9f41-4bec-9146-9ac41e9ad9d4" containerName="istio-proxy" Oct 31 23:24:51.345209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.344683 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1de0a86-4a0b-4429-9088-6f6bf329b42c" containerName="revoke-keys" Oct 31 23:24:51.371484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.371385 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:51.403418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.403342 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp"] Oct 31 23:24:51.403855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.403832 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.408323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.408302 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:24:51.408482 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.408463 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:24:51.472565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.472508 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:51.472976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.472622 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:51.485444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.485358 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.485619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.485510 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.485677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.485663 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p55qz\" (UniqueName: \"kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.586876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.586842 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.587043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.586911 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.587043 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.586986 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p55qz\" (UniqueName: \"kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.587611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.587587 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.589181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.589155 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.599531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.599498 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p55qz\" (UniqueName: \"kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz\") pod \"maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.732692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.732655 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:51.940523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:51.940489 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160baa62-7a99-44ef-ac94-58f851fbc7a5" path="/var/lib/kubelet/pods/160baa62-7a99-44ef-ac94-58f851fbc7a5/volumes" Oct 31 23:24:52.040332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:52.040301 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp"] Oct 31 23:24:52.040741 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:52.040711 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6817c63c_78d4_49d7_af52_6dad8011a066.slice/crio-44e11bdd2594e7caa6ffb30c5af42f52fc12112516e743c72969520912cb2f97 WatchSource:0}: Error finding container 44e11bdd2594e7caa6ffb30c5af42f52fc12112516e743c72969520912cb2f97: Status 404 returned error can't find the container with id 44e11bdd2594e7caa6ffb30c5af42f52fc12112516e743c72969520912cb2f97 Oct 31 23:24:52.410091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:52.410001 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:52.499081 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:52.499044 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" event={"ID":"6817c63c-78d4-49d7-af52-6dad8011a066","Type":"ContainerStarted","Data":"35eb9d5d236cd699edf3e7e6744d487d9eb71d95d968380e0a6c991f0c2673fd"} Oct 31 23:24:52.499570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:52.499109 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" event={"ID":"6817c63c-78d4-49d7-af52-6dad8011a066","Type":"ContainerStarted","Data":"44e11bdd2594e7caa6ffb30c5af42f52fc12112516e743c72969520912cb2f97"} Oct 31 23:24:52.521106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:52.521056 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" podStartSLOduration=1.521043197 podStartE2EDuration="1.521043197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:24:52.517184952 +0000 UTC m=+2118.746739364" watchObservedRunningTime="2026-10-31 23:24:52.521043197 +0000 UTC m=+2118.750597608" Oct 31 23:24:53.232072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:53.232033 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:53.508171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:53.508137 5043 generic.go:356] "Generic (PLEG): container finished" podID="6817c63c-78d4-49d7-af52-6dad8011a066" containerID="35eb9d5d236cd699edf3e7e6744d487d9eb71d95d968380e0a6c991f0c2673fd" exitCode=0 Oct 31 23:24:53.508633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:53.508186 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" event={"ID":"6817c63c-78d4-49d7-af52-6dad8011a066","Type":"ContainerDied","Data":"35eb9d5d236cd699edf3e7e6744d487d9eb71d95d968380e0a6c991f0c2673fd"} Oct 31 23:24:53.942430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:53.942344 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:54.398708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.398670 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:54.532735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.532697 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:54.762171 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.762132 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:54.825468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.825440 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:54.954135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.954099 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca\") pod \"6817c63c-78d4-49d7-af52-6dad8011a066\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " Oct 31 23:24:54.954311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.954162 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz\" (UniqueName: \"kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz\") pod \"6817c63c-78d4-49d7-af52-6dad8011a066\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " Oct 31 23:24:54.954439 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.954424 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca\") pod \"6817c63c-78d4-49d7-af52-6dad8011a066\" (UID: \"6817c63c-78d4-49d7-af52-6dad8011a066\") " Oct 31 23:24:54.954823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.954791 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca" pod "6817c63c-78d4-49d7-af52-6dad8011a066" (UID: "6817c63c-78d4-49d7-af52-6dad8011a066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:24:54.955070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.955051 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:54.956642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.956616 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca" pod "6817c63c-78d4-49d7-af52-6dad8011a066" (UID: "6817c63c-78d4-49d7-af52-6dad8011a066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:24:54.956755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:54.956681 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz" pod "6817c63c-78d4-49d7-af52-6dad8011a066" (UID: "6817c63c-78d4-49d7-af52-6dad8011a066"). InnerVolumeSpecName "kube-api-access-p55qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:24:55.056581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.056490 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6817c63c-78d4-49d7-af52-6dad8011a066-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:55.056581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.056522 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p55qz\" (UniqueName: \"kubernetes.io/projected/6817c63c-78d4-49d7-af52-6dad8011a066-kube-api-access-p55qz\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:24:55.475917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.475820 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:55.527054 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.527023 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" Oct 31 23:24:55.527221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.527070 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp" event={"ID":"6817c63c-78d4-49d7-af52-6dad8011a066","Type":"ContainerDied","Data":"44e11bdd2594e7caa6ffb30c5af42f52fc12112516e743c72969520912cb2f97"} Oct 31 23:24:55.527221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.527153 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e11bdd2594e7caa6ffb30c5af42f52fc12112516e743c72969520912cb2f97" Oct 31 23:24:55.599740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.585768 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:24:55.620108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.620065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:55.855874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:55.855837 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:56.036904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.036858 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:56.188679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.188558 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6"] Oct 31 23:24:56.195415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.195381 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5"] Oct 31 23:24:56.208808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.208765 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6"] Oct 31 23:24:56.235936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.235597 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj"] Oct 31 23:24:56.314064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.313989 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:56.453676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.453584 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:56.606720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.606685 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:56.875361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:56.875325 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.089943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.089894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.266710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.266652 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.418570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.408603 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.574739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.572408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.584344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.584288 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2"] Oct 31 23:24:57.591636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.591604 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c4d32e-spmm2"] Oct 31 23:24:57.712944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.712911 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.744359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.744310 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs"] Oct 31 23:24:57.744671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.744610 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" containerID="cri-o://60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50" gracePeriod=30 Oct 31 23:24:57.846427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.846341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:57.937574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:57.937509 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc995810-97ff-47a0-aea1-1a4b76f4b7d4" path="/var/lib/kubelet/pods/cc995810-97ff-47a0-aea1-1a4b76f4b7d4/volumes" Oct 31 23:24:58.155860 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.155778 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:58.550015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.549980 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:58.690267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.690232 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm"] Oct 31 23:24:58.699618 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.699589 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2"] Oct 31 23:24:58.700287 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.700267 5043 cpu_manager.go:414] "containerMap: removing container" podUID="6817c63c-78d4-49d7-af52-6dad8011a066" containerName="revoke-keys" Oct 31 23:24:58.700384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.700289 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6817c63c-78d4-49d7-af52-6dad8011a066" containerName="revoke-keys" Oct 31 23:24:58.700477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.700463 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6817c63c-78d4-49d7-af52-6dad8011a066" containerName="revoke-keys" Oct 31 23:24:58.703791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.703769 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.703914 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.703843 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9"] Oct 31 23:24:58.710459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.710411 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:24:58.711511 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.711488 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-135248-gj8dm"] Oct 31 23:24:58.711694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.711625 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.713102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.713081 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:24:58.713644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.713093 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:24:58.713644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.713295 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:24:58.714480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.714457 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9"] Oct 31 23:24:58.743477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.743442 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:58.746117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.746091 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2"] Oct 31 23:24:58.852660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.852577 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.852805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.852686 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5h5\" (UniqueName: \"kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.852805 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.852779 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkqf\" (UniqueName: \"kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.852946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.852924 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.853001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.852977 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.853094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.853079 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.887924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.887890 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:58.967288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.967238 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.967495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.967308 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.967495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.967376 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.967495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.967448 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.967713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.967500 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5h5\" (UniqueName: \"kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.967713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.967627 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkqf\" (UniqueName: \"kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:58.980834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.980805 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:58.981897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:58.981867 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:59.007599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.007502 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:59.008973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.008943 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:59.018583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.016091 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5h5\" (UniqueName: \"kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5\") pod \"maas-api-revoke-sub-a1ce14fe-pg7h9\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:59.027719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.026991 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkqf\" (UniqueName: \"kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf\") pod \"maas-api-revoke-sub-6f1dd1cb-jl4c2\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:59.034216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.034173 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:59.048105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.047852 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:24:59.053608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.053585 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:24:59.198733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.198699 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6"] Oct 31 23:24:59.215585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.210014 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7d7eed-wqth6"] Oct 31 23:24:59.263101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.263064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:59.410425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.410343 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:59.418078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.418050 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2"] Oct 31 23:24:59.418456 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:59.418435 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30481cd9_96bc_4880_836d_aa92778ce043.slice/crio-01559166bae85caa76a616e003684d06b23eaba3addd7a12b0949325d144fffb WatchSource:0}: Error finding container 01559166bae85caa76a616e003684d06b23eaba3addd7a12b0949325d144fffb: Status 404 returned error can't find the container with id 01559166bae85caa76a616e003684d06b23eaba3addd7a12b0949325d144fffb Oct 31 23:24:59.473905 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:24:59.473877 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4347d5_4a34_4803_9221_7cc587a14089.slice/crio-8db2f36d369c327f610c66f0e94b542091ec3eb4105ea03290d83cbc4405c3b9 WatchSource:0}: Error finding container 8db2f36d369c327f610c66f0e94b542091ec3eb4105ea03290d83cbc4405c3b9: Status 404 returned error can't find the container with id 8db2f36d369c327f610c66f0e94b542091ec3eb4105ea03290d83cbc4405c3b9 Oct 31 23:24:59.474816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.474787 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9"] Oct 31 23:24:59.674466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.674378 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" event={"ID":"7b4347d5-4a34-4803-9221-7cc587a14089","Type":"ContainerStarted","Data":"8db2f36d369c327f610c66f0e94b542091ec3eb4105ea03290d83cbc4405c3b9"} Oct 31 23:24:59.675787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.675763 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" event={"ID":"30481cd9-96bc-4880-836d-aa92778ce043","Type":"ContainerStarted","Data":"01559166bae85caa76a616e003684d06b23eaba3addd7a12b0949325d144fffb"} Oct 31 23:24:59.797138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.797096 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:24:59.937406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.937379 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53ab05c-86a3-42da-94ef-c520e1227ae8" path="/var/lib/kubelet/pods/b53ab05c-86a3-42da-94ef-c520e1227ae8/volumes" Oct 31 23:24:59.937899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.937875 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df60d0ba-d83d-47bc-b107-7b72e664e26e" path="/var/lib/kubelet/pods/df60d0ba-d83d-47bc-b107-7b72e664e26e/volumes" Oct 31 23:24:59.940502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:24:59.940463 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:00.080404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.080353 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:00.213458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.213381 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:00.686976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.686934 5043 generic.go:356] "Generic (PLEG): container finished" podID="7b4347d5-4a34-4803-9221-7cc587a14089" containerID="c93e12280c32555c8868dcdfbbbce7e7f47da49c88c5b18a0f667833cca27eb6" exitCode=0 Oct 31 23:25:00.687188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.687012 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" event={"ID":"7b4347d5-4a34-4803-9221-7cc587a14089","Type":"ContainerDied","Data":"c93e12280c32555c8868dcdfbbbce7e7f47da49c88c5b18a0f667833cca27eb6"} Oct 31 23:25:00.688824 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.688800 5043 generic.go:356] "Generic (PLEG): container finished" podID="30481cd9-96bc-4880-836d-aa92778ce043" containerID="c028591efecf02c157e8a62a35ec58a357888c5fc982831f138315c029df0a1b" exitCode=0 Oct 31 23:25:00.688953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.688887 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" event={"ID":"30481cd9-96bc-4880-836d-aa92778ce043","Type":"ContainerDied","Data":"c028591efecf02c157e8a62a35ec58a357888c5fc982831f138315c029df0a1b"} Oct 31 23:25:00.928046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:00.927950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:01.189441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.189337 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" containerID="cri-o://91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a" gracePeriod=25 Oct 31 23:25:01.209536 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.209502 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" containerID="cri-o://599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f" gracePeriod=25 Oct 31 23:25:01.498069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.497665 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:25:01.507495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.507462 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276\" (UniqueName: \"kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276\") pod \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " Oct 31 23:25:01.507701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.507537 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume\") pod \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\" (UID: \"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc\") " Oct 31 23:25:01.508454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.508423 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume" pod "3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" (UID: "3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:01.536041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.535075 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276" pod "3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" (UID: "3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc"). InnerVolumeSpecName "kube-api-access-fd276". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:01.549866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.549822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:01.554678 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.554648 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:25:01.608709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.608672 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume\") pod \"ac86da50-1a95-4bd1-adda-981f33721b07\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " Oct 31 23:25:01.609240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.609216 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume" pod "ac86da50-1a95-4bd1-adda-981f33721b07" (UID: "ac86da50-1a95-4bd1-adda-981f33721b07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:01.610089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.609707 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9\" (UniqueName: \"kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9\") pod \"ac86da50-1a95-4bd1-adda-981f33721b07\" (UID: \"ac86da50-1a95-4bd1-adda-981f33721b07\") " Oct 31 23:25:01.611751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.611539 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:01.611869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.611758 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac86da50-1a95-4bd1-adda-981f33721b07-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:01.611869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.611775 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd276\" (UniqueName: \"kubernetes.io/projected/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc-kube-api-access-fd276\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:01.616625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.616587 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9" pod "ac86da50-1a95-4bd1-adda-981f33721b07" (UID: "ac86da50-1a95-4bd1-adda-981f33721b07"). InnerVolumeSpecName "kube-api-access-cpgn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:01.703057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.701153 5043 generic.go:356] "Generic (PLEG): container finished" podID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerID="91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a" exitCode=0 Oct 31 23:25:01.703057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.701240 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" Oct 31 23:25:01.703057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.701516 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" event={"ID":"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc","Type":"ContainerDied","Data":"91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a"} Oct 31 23:25:01.703057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.701604 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6" event={"ID":"3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc","Type":"ContainerDied","Data":"2660259cf70a704b801e297c0a8760deca3fb1f72528cab3932568b5bba83c5a"} Oct 31 23:25:01.703057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.701633 5043 scope.go:122] "RemoveContainer" containerID="91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a" Oct 31 23:25:01.729635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.726228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.730609 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt"] Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.731468 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.731489 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.731514 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.731523 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.731766 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" containerName="payload-processing" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.731781 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" containerName="payload-pre-processing" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.735820 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpgn9\" (UniqueName: \"kubernetes.io/projected/ac86da50-1a95-4bd1-adda-981f33721b07-kube-api-access-cpgn9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.748241 5043 generic.go:356] "Generic (PLEG): container finished" podID="ac86da50-1a95-4bd1-adda-981f33721b07" containerID="599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f" exitCode=0 Oct 31 23:25:01.761660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.748604 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" Oct 31 23:25:01.821940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.821016 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" event={"ID":"ac86da50-1a95-4bd1-adda-981f33721b07","Type":"ContainerDied","Data":"599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f"} Oct 31 23:25:01.821940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.821069 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6" event={"ID":"ac86da50-1a95-4bd1-adda-981f33721b07","Type":"ContainerDied","Data":"08ab60ce4e45cb6ad0f8777706537aa686b81b891e3844901232fefdf2799bf9"} Oct 31 23:25:01.821940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.821165 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.844943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.844902 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s6n\" (UniqueName: \"kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.845250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.845227 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.845611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.845589 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.846326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.846295 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt"] Oct 31 23:25:01.884531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.884477 5043 scope.go:122] "RemoveContainer" containerID="91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a" Oct 31 23:25:01.902993 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:01.902932 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a\": container with ID starting with 91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a not found: ID does not exist" containerID="91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a" Oct 31 23:25:01.903837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.903800 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a"} err="failed to get container status \"91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a\": rpc error: code = NotFound desc = could not find container \"91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a\": container with ID starting with 91b12b9a2dfc92e414997a28ce67ebeac13a7ef8f8f062863f8045f7057dc02a not found: ID does not exist" Oct 31 23:25:01.903991 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.903976 5043 scope.go:122] "RemoveContainer" containerID="599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f" Oct 31 23:25:01.919080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.919048 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6"] Oct 31 23:25:01.937759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.937737 5043 scope.go:122] "RemoveContainer" containerID="599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f" Oct 31 23:25:01.938529 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:01.938508 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f\": container with ID starting with 599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f not found: ID does not exist" containerID="599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f" Oct 31 23:25:01.938733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.938708 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f"} err="failed to get container status \"599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f\": rpc error: code = NotFound desc = could not find container \"599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f\": container with ID starting with 599238fbd67d6fe179eb2fd5efd5930416365289dcc990cdedb3ca4dbfb6040f not found: ID does not exist" Oct 31 23:25:01.948589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.947309 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.948589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.947501 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s6n\" (UniqueName: \"kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.948589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.947619 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.948820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.948594 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.951084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.951061 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.958403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.958381 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c4d32e-5869cc6dcd-x7mm6"] Oct 31 23:25:01.958759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.958743 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6"] Oct 31 23:25:01.960973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.960952 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c4d32e-5f8ddd5b67-d5gb6"] Oct 31 23:25:01.962248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.962224 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s6n\" (UniqueName: \"kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n\") pod \"maas-api-revoke-sub-23292ab3-5lhpt\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:01.996641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:01.996608 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:02.058955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.058914 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:25:02.103588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.103540 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:25:02.150323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150281 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca\") pod \"30481cd9-96bc-4880-836d-aa92778ce043\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " Oct 31 23:25:02.150531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150421 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5\" (UniqueName: \"kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5\") pod \"7b4347d5-4a34-4803-9221-7cc587a14089\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " Oct 31 23:25:02.150531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150454 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca\") pod \"7b4347d5-4a34-4803-9221-7cc587a14089\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " Oct 31 23:25:02.150697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150536 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca\") pod \"30481cd9-96bc-4880-836d-aa92778ce043\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " Oct 31 23:25:02.150697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150591 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca\") pod \"7b4347d5-4a34-4803-9221-7cc587a14089\" (UID: \"7b4347d5-4a34-4803-9221-7cc587a14089\") " Oct 31 23:25:02.150697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150630 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf\" (UniqueName: \"kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf\") pod \"30481cd9-96bc-4880-836d-aa92778ce043\" (UID: \"30481cd9-96bc-4880-836d-aa92778ce043\") " Oct 31 23:25:02.150878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150696 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca" pod "30481cd9-96bc-4880-836d-aa92778ce043" (UID: "30481cd9-96bc-4880-836d-aa92778ce043"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:02.150878 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.150868 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca" pod "7b4347d5-4a34-4803-9221-7cc587a14089" (UID: "7b4347d5-4a34-4803-9221-7cc587a14089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:02.151235 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.151209 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.151336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.151240 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.151834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.151809 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:02.153125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.153095 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca" pod "30481cd9-96bc-4880-836d-aa92778ce043" (UID: "30481cd9-96bc-4880-836d-aa92778ce043"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:02.153378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.153350 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca" pod "7b4347d5-4a34-4803-9221-7cc587a14089" (UID: "7b4347d5-4a34-4803-9221-7cc587a14089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:02.153434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.153364 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5" pod "7b4347d5-4a34-4803-9221-7cc587a14089" (UID: "7b4347d5-4a34-4803-9221-7cc587a14089"). InnerVolumeSpecName "kube-api-access-6g5h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:02.153830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.153806 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf" pod "30481cd9-96bc-4880-836d-aa92778ce043" (UID: "30481cd9-96bc-4880-836d-aa92778ce043"). InnerVolumeSpecName "kube-api-access-tvkqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:02.252231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.252114 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30481cd9-96bc-4880-836d-aa92778ce043-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.252231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.252142 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4347d5-4a34-4803-9221-7cc587a14089-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.252231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.252153 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkqf\" (UniqueName: \"kubernetes.io/projected/30481cd9-96bc-4880-836d-aa92778ce043-kube-api-access-tvkqf\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.252231 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.252163 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5h5\" (UniqueName: \"kubernetes.io/projected/7b4347d5-4a34-4803-9221-7cc587a14089-kube-api-access-6g5h5\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:02.478698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.478656 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt"] Oct 31 23:25:02.479858 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:25:02.479825 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bbccd8_dcde_45ab_9522_56a2b73b7218.slice/crio-c304b26af998e4931258e4d565278f2130d29b239b1d923194ab8d9276cea721 WatchSource:0}: Error finding container c304b26af998e4931258e4d565278f2130d29b239b1d923194ab8d9276cea721: Status 404 returned error can't find the container with id c304b26af998e4931258e4d565278f2130d29b239b1d923194ab8d9276cea721 Oct 31 23:25:02.763255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.763220 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" Oct 31 23:25:02.763453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.763221 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9" event={"ID":"7b4347d5-4a34-4803-9221-7cc587a14089","Type":"ContainerDied","Data":"8db2f36d369c327f610c66f0e94b542091ec3eb4105ea03290d83cbc4405c3b9"} Oct 31 23:25:02.763453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.763351 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db2f36d369c327f610c66f0e94b542091ec3eb4105ea03290d83cbc4405c3b9" Oct 31 23:25:02.765904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.765877 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" Oct 31 23:25:02.766103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.765898 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2" event={"ID":"30481cd9-96bc-4880-836d-aa92778ce043","Type":"ContainerDied","Data":"01559166bae85caa76a616e003684d06b23eaba3addd7a12b0949325d144fffb"} Oct 31 23:25:02.766103 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.765945 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01559166bae85caa76a616e003684d06b23eaba3addd7a12b0949325d144fffb" Oct 31 23:25:02.767345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.767323 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" event={"ID":"12bbccd8-dcde-45ab-9522-56a2b73b7218","Type":"ContainerStarted","Data":"c304b26af998e4931258e4d565278f2130d29b239b1d923194ab8d9276cea721"} Oct 31 23:25:02.901294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:02.901263 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:03.050823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.050796 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:25:03.171560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171518 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.171742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171623 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.171742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171682 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.171881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171756 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.171939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171896 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:03.171939 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171907 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.172048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.171967 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.172048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172015 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.172163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172060 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.172163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172078 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:03.172163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172092 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token\") pod \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\" (UID: \"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f\") " Oct 31 23:25:03.172163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172139 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:03.172376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172298 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:03.172660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172598 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:03.172784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172728 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.172784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172747 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.172784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172756 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.172784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172767 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.172784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.172775 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.174332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.174302 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:03.174432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.174358 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "kube-api-access-c8gl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:03.174432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.174371 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:03.174432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.174377 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo" pod "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" (UID: "9ab0e6eb-0554-4f05-ad1b-06d5faa1550f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:03.274330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.274295 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.274330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.274327 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gl9\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-kube-api-access-c8gl9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.274330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.274337 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.274633 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.274351 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:03.610935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.610895 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:03.777293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.777254 5043 generic.go:356] "Generic (PLEG): container finished" podID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerID="60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50" exitCode=0 Oct 31 23:25:03.777457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.777316 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" Oct 31 23:25:03.777457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.777347 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" event={"ID":"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f","Type":"ContainerDied","Data":"60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50"} Oct 31 23:25:03.777457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.777403 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs" event={"ID":"9ab0e6eb-0554-4f05-ad1b-06d5faa1550f","Type":"ContainerDied","Data":"4f8d7e4668aa540ed494ae32b4d84f23d4cd7a55993fed7ebf05e7b6eec2d93a"} Oct 31 23:25:03.777457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.777423 5043 scope.go:122] "RemoveContainer" containerID="60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50" Oct 31 23:25:03.779375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.779352 5043 generic.go:356] "Generic (PLEG): container finished" podID="12bbccd8-dcde-45ab-9522-56a2b73b7218" containerID="ae9f226a4cabbb473059c912470c648ecc325ab6225c86393f07ce7146bde875" exitCode=0 Oct 31 23:25:03.779493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.779408 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" event={"ID":"12bbccd8-dcde-45ab-9522-56a2b73b7218","Type":"ContainerDied","Data":"ae9f226a4cabbb473059c912470c648ecc325ab6225c86393f07ce7146bde875"} Oct 31 23:25:03.789490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.789466 5043 scope.go:122] "RemoveContainer" containerID="60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50" Oct 31 23:25:03.789778 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:03.789759 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50\": container with ID starting with 60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50 not found: ID does not exist" containerID="60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50" Oct 31 23:25:03.789842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.789789 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50"} err="failed to get container status \"60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50\": rpc error: code = NotFound desc = could not find container \"60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50\": container with ID starting with 60bd4c73bb0ca6b60097bbfb9d167751ffc4286dd350b884b218bbf7d4704e50 not found: ID does not exist" Oct 31 23:25:03.821485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.821454 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs"] Oct 31 23:25:03.825881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.825855 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-ts6rs"] Oct 31 23:25:03.938561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.938521 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc" path="/var/lib/kubelet/pods/3a7b3cf1-7cb6-4997-976d-8f8c3d0773bc/volumes" Oct 31 23:25:03.939091 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.939068 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" path="/var/lib/kubelet/pods/9ab0e6eb-0554-4f05-ad1b-06d5faa1550f/volumes" Oct 31 23:25:03.939510 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:03.939489 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac86da50-1a95-4bd1-adda-981f33721b07" path="/var/lib/kubelet/pods/ac86da50-1a95-4bd1-adda-981f33721b07/volumes" Oct 31 23:25:04.067292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.067220 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" containerID="cri-o://176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc" gracePeriod=45 Oct 31 23:25:04.074088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.074047 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" containerID="cri-o://5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619" gracePeriod=45 Oct 31 23:25:04.200739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.200708 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": dial tcp 10.132.0.88:8000: connect: connection refused" Oct 31 23:25:04.244255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.244140 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Oct 31 23:25:04.316857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.316818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:04.413489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.413461 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:25:04.419831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.419771 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:25:04.490278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490238 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home\") pod \"985fa972-caf5-46b0-be61-ae1fabb59bfe\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " Oct 31 23:25:04.490434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490309 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs\") pod \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " Oct 31 23:25:04.490434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490347 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p\" (UniqueName: \"kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p\") pod \"985fa972-caf5-46b0-be61-ae1fabb59bfe\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " Oct 31 23:25:04.490434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490382 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir\") pod \"985fa972-caf5-46b0-be61-ae1fabb59bfe\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " Oct 31 23:25:04.490434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490419 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n\" (UniqueName: \"kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n\") pod \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " Oct 31 23:25:04.490605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490490 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache\") pod \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " Oct 31 23:25:04.490605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490516 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm\") pod \"985fa972-caf5-46b0-be61-ae1fabb59bfe\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " Oct 31 23:25:04.490605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490574 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache\") pod \"985fa972-caf5-46b0-be61-ae1fabb59bfe\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " Oct 31 23:25:04.490748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490637 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs\") pod \"985fa972-caf5-46b0-be61-ae1fabb59bfe\" (UID: \"985fa972-caf5-46b0-be61-ae1fabb59bfe\") " Oct 31 23:25:04.490748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490704 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir\") pod \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " Oct 31 23:25:04.490748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490731 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm\") pod \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " Oct 31 23:25:04.490837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.490799 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home\") pod \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\" (UID: \"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5\") " Oct 31 23:25:04.491182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.491152 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache" pod "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" (UID: "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.491459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.491435 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home" pod "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" (UID: "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.491497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.491474 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.491769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.491747 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home" pod "985fa972-caf5-46b0-be61-ae1fabb59bfe" (UID: "985fa972-caf5-46b0-be61-ae1fabb59bfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.492090 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.492066 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir" pod "985fa972-caf5-46b0-be61-ae1fabb59bfe" (UID: "985fa972-caf5-46b0-be61-ae1fabb59bfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.492339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.492312 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir" pod "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" (UID: "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.493082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.493055 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache" pod "985fa972-caf5-46b0-be61-ae1fabb59bfe" (UID: "985fa972-caf5-46b0-be61-ae1fabb59bfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.495067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.495027 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm" pod "985fa972-caf5-46b0-be61-ae1fabb59bfe" (UID: "985fa972-caf5-46b0-be61-ae1fabb59bfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.495183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.495076 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p" pod "985fa972-caf5-46b0-be61-ae1fabb59bfe" (UID: "985fa972-caf5-46b0-be61-ae1fabb59bfe"). InnerVolumeSpecName "kube-api-access-mnb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:04.495845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.495824 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs" pod "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" (UID: "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:04.496795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.496767 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n" pod "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" (UID: "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5"). InnerVolumeSpecName "kube-api-access-x2v8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:04.497397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.497371 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm" pod "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" (UID: "2d2bc5af-64a6-4051-8f0a-ed890a14a7f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:04.497531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.497510 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs" pod "985fa972-caf5-46b0-be61-ae1fabb59bfe" (UID: "985fa972-caf5-46b0-be61-ae1fabb59bfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:04.592771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592734 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.592771 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592770 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592781 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb4p\" (UniqueName: \"kubernetes.io/projected/985fa972-caf5-46b0-be61-ae1fabb59bfe-kube-api-access-mnb4p\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592793 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592802 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2v8n\" (UniqueName: \"kubernetes.io/projected/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-kube-api-access-x2v8n\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592810 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592819 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/985fa972-caf5-46b0-be61-ae1fabb59bfe-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592827 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/985fa972-caf5-46b0-be61-ae1fabb59bfe-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592835 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592844 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.593004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.592852 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:04.789642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.789605 5043 generic.go:356] "Generic (PLEG): container finished" podID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerID="176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc" exitCode=0 Oct 31 23:25:04.789803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.789658 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" Oct 31 23:25:04.789803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.789699 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" event={"ID":"985fa972-caf5-46b0-be61-ae1fabb59bfe","Type":"ContainerDied","Data":"176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc"} Oct 31 23:25:04.789803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.789742 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8" event={"ID":"985fa972-caf5-46b0-be61-ae1fabb59bfe","Type":"ContainerDied","Data":"125a33e7d3b3898cdc07abca07707397dbc1e1e210ec7ac73f6624416e93475f"} Oct 31 23:25:04.789803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.789760 5043 scope.go:122] "RemoveContainer" containerID="176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc" Oct 31 23:25:04.791200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.791168 5043 generic.go:356] "Generic (PLEG): container finished" podID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerID="5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619" exitCode=0 Oct 31 23:25:04.791341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.791216 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" Oct 31 23:25:04.791341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.791260 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" event={"ID":"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5","Type":"ContainerDied","Data":"5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619"} Oct 31 23:25:04.791341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.791299 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z" event={"ID":"2d2bc5af-64a6-4051-8f0a-ed890a14a7f5","Type":"ContainerDied","Data":"00d2023a6256168218bfc3eff55824130dd642fdb698f24a3a0f6d17fe2dfd68"} Oct 31 23:25:04.817710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.817645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:04.832255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.832146 5043 scope.go:122] "RemoveContainer" containerID="176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc" Oct 31 23:25:04.833414 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:04.833385 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc\": container with ID starting with 176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc not found: ID does not exist" containerID="176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc" Oct 31 23:25:04.833589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.833563 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc"} err="failed to get container status \"176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc\": rpc error: code = NotFound desc = could not find container \"176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc\": container with ID starting with 176d5919e78b58573b9da498e3d7df7c8d9721b5509a60e3faeb9e9bb3e155bc not found: ID does not exist" Oct 31 23:25:04.833710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.833695 5043 scope.go:122] "RemoveContainer" containerID="5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619" Oct 31 23:25:04.840472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.840447 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8"] Oct 31 23:25:04.851071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.851039 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/distinct-w3-67fce9-kserve-6bcd4544fb-rh4f8"] Oct 31 23:25:04.861632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.857264 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z"] Oct 31 23:25:04.870566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.865813 5043 scope.go:122] "RemoveContainer" containerID="5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619" Oct 31 23:25:04.886812 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:04.883007 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619\": container with ID starting with 5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619 not found: ID does not exist" containerID="5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619" Oct 31 23:25:04.886812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.883053 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619"} err="failed to get container status \"5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619\": rpc error: code = NotFound desc = could not find container \"5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619\": container with ID starting with 5568f038cf734c638994c684a8805a81bb877b80e762bc000a65a23e2a964619 not found: ID does not exist" Oct 31 23:25:04.911174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.911045 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-67fce9/premium-w3-67fce9-kserve-559687cd89-bqv4z"] Oct 31 23:25:04.955567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:04.955328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:05.103579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.103442 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:05.104072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.103868 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk"] Oct 31 23:25:05.104193 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.104162 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" containerID="cri-o://a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c" gracePeriod=30 Oct 31 23:25:05.112352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.112325 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca\") pod \"12bbccd8-dcde-45ab-9522-56a2b73b7218\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " Oct 31 23:25:05.112463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.112400 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n\" (UniqueName: \"kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n\") pod \"12bbccd8-dcde-45ab-9522-56a2b73b7218\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " Oct 31 23:25:05.112569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.112541 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca\") pod \"12bbccd8-dcde-45ab-9522-56a2b73b7218\" (UID: \"12bbccd8-dcde-45ab-9522-56a2b73b7218\") " Oct 31 23:25:05.113489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.113465 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca" pod "12bbccd8-dcde-45ab-9522-56a2b73b7218" (UID: "12bbccd8-dcde-45ab-9522-56a2b73b7218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:05.116466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.116439 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca" pod "12bbccd8-dcde-45ab-9522-56a2b73b7218" (UID: "12bbccd8-dcde-45ab-9522-56a2b73b7218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:05.120101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.120074 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n" pod "12bbccd8-dcde-45ab-9522-56a2b73b7218" (UID: "12bbccd8-dcde-45ab-9522-56a2b73b7218"). InnerVolumeSpecName "kube-api-access-p9s6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:05.142430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.135483 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:05.219444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.214083 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:05.219444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.214142 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bbccd8-dcde-45ab-9522-56a2b73b7218-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:05.219444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.214176 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s6n\" (UniqueName: \"kubernetes.io/projected/12bbccd8-dcde-45ab-9522-56a2b73b7218-kube-api-access-p9s6n\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:05.236394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.236364 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:25:05.281922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.281739 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:05.316802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.315624 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume\") pod \"65dd98b9-8631-405f-88bc-f7a4e090177a\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " Oct 31 23:25:05.316802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.315786 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79\" (UniqueName: \"kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79\") pod \"65dd98b9-8631-405f-88bc-f7a4e090177a\" (UID: \"65dd98b9-8631-405f-88bc-f7a4e090177a\") " Oct 31 23:25:05.317532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.317503 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume" pod "65dd98b9-8631-405f-88bc-f7a4e090177a" (UID: "65dd98b9-8631-405f-88bc-f7a4e090177a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:05.329066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.329020 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79" pod "65dd98b9-8631-405f-88bc-f7a4e090177a" (UID: "65dd98b9-8631-405f-88bc-f7a4e090177a"). InnerVolumeSpecName "kube-api-access-hdt79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:05.417134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.417100 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65dd98b9-8631-405f-88bc-f7a4e090177a-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:05.417134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.417130 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdt79\" (UniqueName: \"kubernetes.io/projected/65dd98b9-8631-405f-88bc-f7a4e090177a-kube-api-access-hdt79\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:05.688898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.688819 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn"] Oct 31 23:25:05.689420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689406 5043 cpu_manager.go:414] "containerMap: removing container" podUID="12bbccd8-dcde-45ab-9522-56a2b73b7218" containerName="revoke-keys" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689422 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bbccd8-dcde-45ab-9522-56a2b73b7218" containerName="revoke-keys" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689432 5043 cpu_manager.go:414] "containerMap: removing container" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689437 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689445 5043 cpu_manager.go:414] "containerMap: removing container" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689449 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689459 5043 cpu_manager.go:414] "containerMap: removing container" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" Oct 31 23:25:05.689466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689463 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689475 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7b4347d5-4a34-4803-9221-7cc587a14089" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689480 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4347d5-4a34-4803-9221-7cc587a14089" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689487 5043 cpu_manager.go:414] "containerMap: removing container" podUID="30481cd9-96bc-4880-836d-aa92778ce043" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689492 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30481cd9-96bc-4880-836d-aa92778ce043" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689499 5043 cpu_manager.go:414] "containerMap: removing container" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689503 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689602 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bbccd8-dcde-45ab-9522-56a2b73b7218" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689610 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" containerName="main" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689618 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab0e6eb-0554-4f05-ad1b-06d5faa1550f" containerName="istio-proxy" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689624 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerName="payload-processing" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689631 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4347d5-4a34-4803-9221-7cc587a14089" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689637 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="30481cd9-96bc-4880-836d-aa92778ce043" containerName="revoke-keys" Oct 31 23:25:05.689793 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.689643 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" containerName="main" Oct 31 23:25:05.693250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.693229 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.703525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.703497 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn"] Oct 31 23:25:05.720940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.720904 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm9vr\" (UniqueName: \"kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.721162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.721135 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.721381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.721364 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.802708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.802676 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" Oct 31 23:25:05.802708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.802697 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt" event={"ID":"12bbccd8-dcde-45ab-9522-56a2b73b7218","Type":"ContainerDied","Data":"c304b26af998e4931258e4d565278f2130d29b239b1d923194ab8d9276cea721"} Oct 31 23:25:05.802940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.802759 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c304b26af998e4931258e4d565278f2130d29b239b1d923194ab8d9276cea721" Oct 31 23:25:05.804289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.804260 5043 generic.go:356] "Generic (PLEG): container finished" podID="65dd98b9-8631-405f-88bc-f7a4e090177a" containerID="a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe" exitCode=0 Oct 31 23:25:05.804438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.804301 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" Oct 31 23:25:05.804438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.804340 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" event={"ID":"65dd98b9-8631-405f-88bc-f7a4e090177a","Type":"ContainerDied","Data":"a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe"} Oct 31 23:25:05.804438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.804399 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml" event={"ID":"65dd98b9-8631-405f-88bc-f7a4e090177a","Type":"ContainerDied","Data":"d91d3a59483a95c7a35cedc3406ddb5be23484b379705cfcd397a0d0e0ee6aca"} Oct 31 23:25:05.804438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.804427 5043 scope.go:122] "RemoveContainer" containerID="a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe" Oct 31 23:25:05.818944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.818919 5043 scope.go:122] "RemoveContainer" containerID="a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe" Oct 31 23:25:05.819297 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:05.819272 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe\": container with ID starting with a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe not found: ID does not exist" containerID="a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe" Oct 31 23:25:05.819388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.819303 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe"} err="failed to get container status \"a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe\": rpc error: code = NotFound desc = could not find container \"a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe\": container with ID starting with a716893a72e9f91007fcb13a83a67654f0eb7ef202418438e66eef14dce536fe not found: ID does not exist" Oct 31 23:25:05.823101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.823076 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.823221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.823198 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm9vr\" (UniqueName: \"kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.823282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.823223 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.823718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.823700 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.825855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.825830 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.835224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.835197 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml"] Oct 31 23:25:05.843682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.843659 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7d7eed-654b877cfd-78tml"] Oct 31 23:25:05.844493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.844473 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm9vr\" (UniqueName: \"kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:05.939686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.939572 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2bc5af-64a6-4051-8f0a-ed890a14a7f5" path="/var/lib/kubelet/pods/2d2bc5af-64a6-4051-8f0a-ed890a14a7f5/volumes" Oct 31 23:25:05.940224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.940195 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dd98b9-8631-405f-88bc-f7a4e090177a" path="/var/lib/kubelet/pods/65dd98b9-8631-405f-88bc-f7a4e090177a/volumes" Oct 31 23:25:05.940863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:05.940839 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985fa972-caf5-46b0-be61-ae1fabb59bfe" path="/var/lib/kubelet/pods/985fa972-caf5-46b0-be61-ae1fabb59bfe/volumes" Oct 31 23:25:06.016693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:06.013174 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:06.213389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:06.212252 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:06.360504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:06.360474 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn"] Oct 31 23:25:06.364703 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:25:06.364662 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8d6975_98da_402d_b929_b08fbf891ff6.slice/crio-2baccbc7411a83db8746f72de60626a5375215f6ac4ea32ba87ebc42c4705da2 WatchSource:0}: Error finding container 2baccbc7411a83db8746f72de60626a5375215f6ac4ea32ba87ebc42c4705da2: Status 404 returned error can't find the container with id 2baccbc7411a83db8746f72de60626a5375215f6ac4ea32ba87ebc42c4705da2 Oct 31 23:25:06.848732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:06.848641 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" event={"ID":"5b8d6975-98da-402d-b929-b08fbf891ff6","Type":"ContainerStarted","Data":"7d6e79ea590e48e8ceea61a5fdb36e8fb9158bd133ba9ac87b792a3480d9eb4d"} Oct 31 23:25:06.848732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:06.848714 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" event={"ID":"5b8d6975-98da-402d-b929-b08fbf891ff6","Type":"ContainerStarted","Data":"2baccbc7411a83db8746f72de60626a5375215f6ac4ea32ba87ebc42c4705da2"} Oct 31 23:25:06.874126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:06.874073 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" podStartSLOduration=1.874057273 podStartE2EDuration="1.874057273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:25:06.871888772 +0000 UTC m=+2133.101443185" watchObservedRunningTime="2026-10-31 23:25:06.874057273 +0000 UTC m=+2133.103611721" Oct 31 23:25:07.251382 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.251352 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:07.454826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.454790 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:07.572924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.572866 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb"] Oct 31 23:25:07.573984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.573127 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" containerID="cri-o://0570764552fa2ecedac368b40747d85ec5da96307a5f7a59731701891cd2befa" gracePeriod=30 Oct 31 23:25:07.580034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.580002 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll"] Oct 31 23:25:07.580254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.580230 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" containerID="cri-o://e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2" gracePeriod=30 Oct 31 23:25:07.584971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.584946 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7"] Oct 31 23:25:07.585144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.585123 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" containerID="cri-o://62774c08ba9a484dd0ebc15316de6103802c19591ca8999ef4eeadc376445b46" gracePeriod=30 Oct 31 23:25:07.639900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.639866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:07.793200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.793093 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:07.881458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.880062 5043 generic.go:356] "Generic (PLEG): container finished" podID="5b8d6975-98da-402d-b929-b08fbf891ff6" containerID="7d6e79ea590e48e8ceea61a5fdb36e8fb9158bd133ba9ac87b792a3480d9eb4d" exitCode=0 Oct 31 23:25:07.881458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:07.880334 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" event={"ID":"5b8d6975-98da-402d-b929-b08fbf891ff6","Type":"ContainerDied","Data":"7d6e79ea590e48e8ceea61a5fdb36e8fb9158bd133ba9ac87b792a3480d9eb4d"} Oct 31 23:25:08.300246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:08.300196 5043 patch_prober.go:29] interesting pod/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:08.300809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:08.300262 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:08.331367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:08.331329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:08.935187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:08.934630 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.084390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.084344 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.103754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.103730 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:09.215934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.215835 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr\" (UniqueName: \"kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr\") pod \"5b8d6975-98da-402d-b929-b08fbf891ff6\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " Oct 31 23:25:09.220331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.220213 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca\") pod \"5b8d6975-98da-402d-b929-b08fbf891ff6\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " Oct 31 23:25:09.220627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.220392 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca\") pod \"5b8d6975-98da-402d-b929-b08fbf891ff6\" (UID: \"5b8d6975-98da-402d-b929-b08fbf891ff6\") " Oct 31 23:25:09.241184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.241122 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca" pod "5b8d6975-98da-402d-b929-b08fbf891ff6" (UID: "5b8d6975-98da-402d-b929-b08fbf891ff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:09.241479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.241421 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:09.241637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.241587 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf"] Oct 31 23:25:09.241912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.241860 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" containerID="cri-o://c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d" gracePeriod=30 Oct 31 23:25:09.242046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.241940 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.253441 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.253416 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr"] Oct 31 23:25:09.253686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.253653 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" containerID="cri-o://ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355" gracePeriod=30 Oct 31 23:25:09.264989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.264964 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr" pod "5b8d6975-98da-402d-b929-b08fbf891ff6" (UID: "5b8d6975-98da-402d-b929-b08fbf891ff6"). InnerVolumeSpecName "kube-api-access-hm9vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:09.267864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.267835 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca" pod "5b8d6975-98da-402d-b929-b08fbf891ff6" (UID: "5b8d6975-98da-402d-b929-b08fbf891ff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:09.362570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.347644 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm9vr\" (UniqueName: \"kubernetes.io/projected/5b8d6975-98da-402d-b929-b08fbf891ff6-kube-api-access-hm9vr\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:09.362570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.347684 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b8d6975-98da-402d-b929-b08fbf891ff6-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:09.382566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.381960 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.562945 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.562912 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.700463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.700430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.833906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.833828 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:09.899384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.899356 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" Oct 31 23:25:09.899584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.899401 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn" event={"ID":"5b8d6975-98da-402d-b929-b08fbf891ff6","Type":"ContainerDied","Data":"2baccbc7411a83db8746f72de60626a5375215f6ac4ea32ba87ebc42c4705da2"} Oct 31 23:25:09.899584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.899451 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2baccbc7411a83db8746f72de60626a5375215f6ac4ea32ba87ebc42c4705da2" Oct 31 23:25:09.982567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:09.975741 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:10.111892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.111811 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6"] Oct 31 23:25:10.116689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.116658 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-67fce9-jqbh6"] Oct 31 23:25:10.123016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.122984 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:10.262779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.262728 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:10.408205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.408182 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:25:10.467743 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:10.467708 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:25:10.567247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567212 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567259 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567287 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567342 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567475 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567581 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567593 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:10.567709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567693 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567818 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567855 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket\") pod \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\" (UID: \"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef\") " Oct 31 23:25:10.567967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.567949 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:10.568115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568089 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:10.568205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568124 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:10.568278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568250 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:10.568437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568418 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.568504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568438 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.568504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568447 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.568504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568456 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.568504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.568464 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.569756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.569725 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:10.569837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.569799 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "kube-api-access-gwcqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:10.569874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.569830 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:10.570931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.570911 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy" pod "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" (UID: "3ef7adc6-4f5e-4875-84bc-629aeb4d2eef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:10.631315 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.631219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:10.669676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.669642 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.669676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.669671 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwcqb\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-kube-api-access-gwcqb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.669864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.669689 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.669864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.669703 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:10.910018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.909797 5043 generic.go:356] "Generic (PLEG): container finished" podID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerID="a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c" exitCode=0 Oct 31 23:25:10.910018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.909854 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" event={"ID":"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef","Type":"ContainerDied","Data":"a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c"} Oct 31 23:25:10.910018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.909921 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" event={"ID":"3ef7adc6-4f5e-4875-84bc-629aeb4d2eef","Type":"ContainerDied","Data":"a5bec2a62c23d00f6d2ade32dd7a51cc01f8da18b644c4d3a468723abd01fef8"} Oct 31 23:25:10.910018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.909951 5043 scope.go:122] "RemoveContainer" containerID="a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c" Oct 31 23:25:10.910363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.909868 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk" Oct 31 23:25:10.936076 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.936037 5043 scope.go:122] "RemoveContainer" containerID="5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211" Oct 31 23:25:10.936356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.936329 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:10.949598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.948887 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk"] Oct 31 23:25:10.963430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.963395 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c4d32e-openshift-default-5b657bffd5-d9wjk"] Oct 31 23:25:10.971142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.971118 5043 scope.go:122] "RemoveContainer" containerID="a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c" Oct 31 23:25:10.971509 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:10.971484 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c\": container with ID starting with a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c not found: ID does not exist" containerID="a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c" Oct 31 23:25:10.971599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.971524 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c"} err="failed to get container status \"a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c\": rpc error: code = NotFound desc = could not find container \"a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c\": container with ID starting with a58f68981a7c2bc704c6f54cad6d16e053a7c2d862d399188c3e289a109e096c not found: ID does not exist" Oct 31 23:25:10.971599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.971562 5043 scope.go:122] "RemoveContainer" containerID="5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211" Oct 31 23:25:10.971873 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:10.971853 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211\": container with ID starting with 5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211 not found: ID does not exist" containerID="5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211" Oct 31 23:25:10.971941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:10.971879 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211"} err="failed to get container status \"5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211\": rpc error: code = NotFound desc = could not find container \"5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211\": container with ID starting with 5436bfc81ccbc6d9e32c2b23606ad12b0ca45bd544295f675af2231ab5122211 not found: ID does not exist" Oct 31 23:25:11.227844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.227739 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" containerID="cri-o://4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd" gracePeriod=45 Oct 31 23:25:11.309220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.309173 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" containerID="cri-o://18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd" gracePeriod=45 Oct 31 23:25:11.524426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.522525 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:11.664485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.664128 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:25:11.674762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.674732 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:25:11.789337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789299 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs\" (UniqueName: \"kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs\") pod \"01442497-4a44-42db-bd0d-7d7839a7efe1\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " Oct 31 23:25:11.789528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789380 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home\") pod \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " Oct 31 23:25:11.789528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789493 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache\") pod \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " Oct 31 23:25:11.789720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789534 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z\" (UniqueName: \"kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z\") pod \"01442497-4a44-42db-bd0d-7d7839a7efe1\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " Oct 31 23:25:11.789720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789603 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir\") pod \"01442497-4a44-42db-bd0d-7d7839a7efe1\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " Oct 31 23:25:11.789720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789698 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache\") pod \"01442497-4a44-42db-bd0d-7d7839a7efe1\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " Oct 31 23:25:11.789844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789728 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm\") pod \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " Oct 31 23:25:11.789844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789787 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8\" (UniqueName: \"kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8\") pod \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " Oct 31 23:25:11.789844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789831 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs\") pod \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " Oct 31 23:25:11.789966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789868 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir\") pod \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\" (UID: \"a0f8db39-ca3d-4e82-a6de-774b40cb6325\") " Oct 31 23:25:11.789966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789907 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home\") pod \"01442497-4a44-42db-bd0d-7d7839a7efe1\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " Oct 31 23:25:11.789966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789927 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache" pod "a0f8db39-ca3d-4e82-a6de-774b40cb6325" (UID: "a0f8db39-ca3d-4e82-a6de-774b40cb6325"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.790077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.789967 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm\") pod \"01442497-4a44-42db-bd0d-7d7839a7efe1\" (UID: \"01442497-4a44-42db-bd0d-7d7839a7efe1\") " Oct 31 23:25:11.790145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.790127 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home" pod "a0f8db39-ca3d-4e82-a6de-774b40cb6325" (UID: "a0f8db39-ca3d-4e82-a6de-774b40cb6325"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.790534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.790514 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.790739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.790712 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir" pod "01442497-4a44-42db-bd0d-7d7839a7efe1" (UID: "01442497-4a44-42db-bd0d-7d7839a7efe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.790739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.790729 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.791253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.791228 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache" pod "01442497-4a44-42db-bd0d-7d7839a7efe1" (UID: "01442497-4a44-42db-bd0d-7d7839a7efe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.791349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.791284 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir" pod "a0f8db39-ca3d-4e82-a6de-774b40cb6325" (UID: "a0f8db39-ca3d-4e82-a6de-774b40cb6325"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.791459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.791441 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home" pod "01442497-4a44-42db-bd0d-7d7839a7efe1" (UID: "01442497-4a44-42db-bd0d-7d7839a7efe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.793686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.793656 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm" pod "01442497-4a44-42db-bd0d-7d7839a7efe1" (UID: "01442497-4a44-42db-bd0d-7d7839a7efe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.793814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.793790 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs" pod "a0f8db39-ca3d-4e82-a6de-774b40cb6325" (UID: "a0f8db39-ca3d-4e82-a6de-774b40cb6325"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:11.795849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.795526 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8" pod "a0f8db39-ca3d-4e82-a6de-774b40cb6325" (UID: "a0f8db39-ca3d-4e82-a6de-774b40cb6325"). InnerVolumeSpecName "kube-api-access-8djd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:11.795849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.795698 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm" pod "a0f8db39-ca3d-4e82-a6de-774b40cb6325" (UID: "a0f8db39-ca3d-4e82-a6de-774b40cb6325"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:11.795849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.795743 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z" pod "01442497-4a44-42db-bd0d-7d7839a7efe1" (UID: "01442497-4a44-42db-bd0d-7d7839a7efe1"). InnerVolumeSpecName "kube-api-access-qt88z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:11.796517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.796490 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs" pod "01442497-4a44-42db-bd0d-7d7839a7efe1" (UID: "01442497-4a44-42db-bd0d-7d7839a7efe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:11.891941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.891895 5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-model-cache\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.891941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.891938 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.891941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.891951 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djd8\" (UniqueName: \"kubernetes.io/projected/a0f8db39-ca3d-4e82-a6de-774b40cb6325-kube-api-access-8djd8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.891961 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.891992 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f8db39-ca3d-4e82-a6de-774b40cb6325-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.892000 5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-home\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.892008 5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-dshm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.892016 5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01442497-4a44-42db-bd0d-7d7839a7efe1-tls-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.892024 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt88z\" (UniqueName: \"kubernetes.io/projected/01442497-4a44-42db-bd0d-7d7839a7efe1-kube-api-access-qt88z\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.892291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.892032 5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01442497-4a44-42db-bd0d-7d7839a7efe1-tmp-dir\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:11.922415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.922375 5043 generic.go:356] "Generic (PLEG): container finished" podID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerID="4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd" exitCode=0 Oct 31 23:25:11.922634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.922446 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" Oct 31 23:25:11.922634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.922469 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" event={"ID":"01442497-4a44-42db-bd0d-7d7839a7efe1","Type":"ContainerDied","Data":"4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd"} Oct 31 23:25:11.922634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.922518 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5" event={"ID":"01442497-4a44-42db-bd0d-7d7839a7efe1","Type":"ContainerDied","Data":"faf49198c62aa8df22d378d8a58c35582004f5ee32e9b6ab4c24b6ac0b81d41c"} Oct 31 23:25:11.922634 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.922541 5043 scope.go:122] "RemoveContainer" containerID="4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd" Oct 31 23:25:11.924484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.924461 5043 generic.go:356] "Generic (PLEG): container finished" podID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerID="18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd" exitCode=0 Oct 31 23:25:11.924667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.924645 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" event={"ID":"a0f8db39-ca3d-4e82-a6de-774b40cb6325","Type":"ContainerDied","Data":"18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd"} Oct 31 23:25:11.924746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.924689 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" event={"ID":"a0f8db39-ca3d-4e82-a6de-774b40cb6325","Type":"ContainerDied","Data":"494c1a00eb28d85ae2d0c285ca5efce05149430e767384b4c53f910e1b81ca15"} Oct 31 23:25:11.924801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.924771 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj" Oct 31 23:25:11.939836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.939778 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" path="/var/lib/kubelet/pods/3ef7adc6-4f5e-4875-84bc-629aeb4d2eef/volumes" Oct 31 23:25:11.941269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.941246 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1de0a86-4a0b-4429-9088-6f6bf329b42c" path="/var/lib/kubelet/pods/d1de0a86-4a0b-4429-9088-6f6bf329b42c/volumes" Oct 31 23:25:11.955336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.955315 5043 scope.go:122] "RemoveContainer" containerID="4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd" Oct 31 23:25:11.955742 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:11.955711 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd\": container with ID starting with 4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd not found: ID does not exist" containerID="4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd" Oct 31 23:25:11.955849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.955756 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd"} err="failed to get container status \"4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd\": rpc error: code = NotFound desc = could not find container \"4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd\": container with ID starting with 4d991f5c05c3af22df44bb31e0bcd357815d430707780a43719ac9683f164ecd not found: ID does not exist" Oct 31 23:25:11.955849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.955779 5043 scope.go:122] "RemoveContainer" containerID="18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd" Oct 31 23:25:11.956751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.956730 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5"] Oct 31 23:25:11.962227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.962205 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/distinct-2-w1-04c5c2-kserve-6b7b67c56b-jbsp5"] Oct 31 23:25:11.967744 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.967724 5043 scope.go:122] "RemoveContainer" containerID="18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd" Oct 31 23:25:11.968061 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:11.968041 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd\": container with ID starting with 18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd not found: ID does not exist" containerID="18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd" Oct 31 23:25:11.968134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.968067 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd"} err="failed to get container status \"18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd\": rpc error: code = NotFound desc = could not find container \"18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd\": container with ID starting with 18d33c56b08e5f9af73e75de4aa063919499882c1d205ed697a33b12cd3b1ccd not found: ID does not exist" Oct 31 23:25:11.972884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.972864 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj"] Oct 31 23:25:11.979843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:11.979815 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-04c5c2/simulator-w1-04c5c2-kserve-7f7cf99bbc-d9nzj"] Oct 31 23:25:12.081190 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.081112 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:12.164738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.164707 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp"] Oct 31 23:25:12.168317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.168294 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-04c5c2-58pkp"] Oct 31 23:25:12.284964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.284904 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:12.456407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.456276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:12.696725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.696675 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:12.838767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.838631 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:12.896355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.896327 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:25:12.958953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.958420 5043 generic.go:356] "Generic (PLEG): container finished" podID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerID="0570764552fa2ecedac368b40747d85ec5da96307a5f7a59731701891cd2befa" exitCode=0 Oct 31 23:25:12.958953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.958521 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" event={"ID":"d336aa46-fc0a-4c34-a9dc-a45949939314","Type":"ContainerDied","Data":"0570764552fa2ecedac368b40747d85ec5da96307a5f7a59731701891cd2befa"} Oct 31 23:25:12.961635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.961606 5043 generic.go:356] "Generic (PLEG): container finished" podID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerID="e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2" exitCode=0 Oct 31 23:25:12.961876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.961855 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" event={"ID":"e0d9c0df-ff9e-4bab-8621-ee2537b266de","Type":"ContainerDied","Data":"e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2"} Oct 31 23:25:12.962023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.962009 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" event={"ID":"e0d9c0df-ff9e-4bab-8621-ee2537b266de","Type":"ContainerDied","Data":"4ddc6f267c1b7be9352861d1896d2a8e16a9f1e101b7b88aec34002618a71223"} Oct 31 23:25:12.962200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.962188 5043 scope.go:122] "RemoveContainer" containerID="e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2" Oct 31 23:25:12.962383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.962371 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll" Oct 31 23:25:12.966615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.966292 5043 generic.go:356] "Generic (PLEG): container finished" podID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerID="62774c08ba9a484dd0ebc15316de6103802c19591ca8999ef4eeadc376445b46" exitCode=0 Oct 31 23:25:12.966615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.966516 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" event={"ID":"f5fdaed9-d6a5-4748-b93a-d385a615164a","Type":"ContainerDied","Data":"62774c08ba9a484dd0ebc15316de6103802c19591ca8999ef4eeadc376445b46"} Oct 31 23:25:12.971074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.970784 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:25:12.982209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.981045 5043 scope.go:122] "RemoveContainer" containerID="e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2" Oct 31 23:25:12.982209 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:12.981593 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2\": container with ID starting with e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2 not found: ID does not exist" containerID="e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2" Oct 31 23:25:12.982209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:12.981632 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2"} err="failed to get container status \"e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2\": rpc error: code = NotFound desc = could not find container \"e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2\": container with ID starting with e7059e89304314e102d69742d9daa4b9729e699e5c16e4f063acc309be3a0de2 not found: ID does not exist" Oct 31 23:25:13.010642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.010359 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:25:13.030424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030393 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030515 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030574 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030599 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030632 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030652 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030705 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030753 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.030823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.030795 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket\") pod \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\" (UID: \"e0d9c0df-ff9e-4bab-8621-ee2537b266de\") " Oct 31 23:25:13.031626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.031597 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.031976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.031942 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:13.032840 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.032682 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.039170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.033014 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.042117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.040232 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "kube-api-access-bsjcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.043879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.043739 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.046373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.046346 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:13.048270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.048246 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.054660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.054632 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy" pod "e0d9c0df-ff9e-4bab-8621-ee2537b266de" (UID: "e0d9c0df-ff9e-4bab-8621-ee2537b266de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.132538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132499 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.132538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132556 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.132811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132575 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.132811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132615 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.132811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132725 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.132811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132769 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.133027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132843 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132894 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132970 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132977 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.133027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.132997 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.133291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133059 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.133291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133119 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133160 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133208 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.133291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133237 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133290 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.133576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133321 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket\") pod \"d336aa46-fc0a-4c34-a9dc-a45949939314\" (UID: \"d336aa46-fc0a-4c34-a9dc-a45949939314\") " Oct 31 23:25:13.133576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133354 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.133576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133404 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert\") pod \"f5fdaed9-d6a5-4748-b93a-d385a615164a\" (UID: \"f5fdaed9-d6a5-4748-b93a-d385a615164a\") " Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133963 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133991 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134007 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134023 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsjcg\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-kube-api-access-bsjcg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134037 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134052 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134066 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134080 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134097 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134110 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.134137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134127 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0d9c0df-ff9e-4bab-8621-ee2537b266de-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133235 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133331 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.133582 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134519 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134531 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134827 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.134968 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.135351 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.135755 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.135638 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.136277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.135851 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.136277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.135887 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "kube-api-access-tcmzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.136394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.136330 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "kube-api-access-n2txn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:13.137411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.137384 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:13.137680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.137657 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.137834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.137812 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo" pod "d336aa46-fc0a-4c34-a9dc-a45949939314" (UID: "d336aa46-fc0a-4c34-a9dc-a45949939314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:13.138384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.138363 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy" pod "f5fdaed9-d6a5-4748-b93a-d385a615164a" (UID: "f5fdaed9-d6a5-4748-b93a-d385a615164a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:13.235689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235586 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2txn\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-kube-api-access-n2txn\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.235689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235626 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.235689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235642 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.235689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235657 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.235689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235671 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.235689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235684 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235699 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235713 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235726 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235739 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5fdaed9-d6a5-4748-b93a-d385a615164a-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235752 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235767 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235780 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235794 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmzb\" (UniqueName: \"kubernetes.io/projected/f5fdaed9-d6a5-4748-b93a-d385a615164a-kube-api-access-tcmzb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235807 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d336aa46-fc0a-4c34-a9dc-a45949939314-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.236006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.235820 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d336aa46-fc0a-4c34-a9dc-a45949939314-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:13.289182 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.289150 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll"] Oct 31 23:25:13.293126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.293098 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-tqvll"] Oct 31 23:25:13.513620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.513580 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:13.938478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.938391 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" path="/var/lib/kubelet/pods/01442497-4a44-42db-bd0d-7d7839a7efe1/volumes" Oct 31 23:25:13.938985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.938891 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6817c63c-78d4-49d7-af52-6dad8011a066" path="/var/lib/kubelet/pods/6817c63c-78d4-49d7-af52-6dad8011a066/volumes" Oct 31 23:25:13.939283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.939267 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" path="/var/lib/kubelet/pods/a0f8db39-ca3d-4e82-a6de-774b40cb6325/volumes" Oct 31 23:25:13.940018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.939999 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" path="/var/lib/kubelet/pods/e0d9c0df-ff9e-4bab-8621-ee2537b266de/volumes" Oct 31 23:25:13.979432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.979403 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" Oct 31 23:25:13.979603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.979410 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb" event={"ID":"d336aa46-fc0a-4c34-a9dc-a45949939314","Type":"ContainerDied","Data":"8b141b09db3b33868298427eacbfeefe7c4824043ad86b682ce0e4ac6cc3ed74"} Oct 31 23:25:13.979603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.979578 5043 scope.go:122] "RemoveContainer" containerID="0570764552fa2ecedac368b40747d85ec5da96307a5f7a59731701891cd2befa" Oct 31 23:25:13.982646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.982626 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" Oct 31 23:25:13.982774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.982677 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7" event={"ID":"f5fdaed9-d6a5-4748-b93a-d385a615164a","Type":"ContainerDied","Data":"523642639beff6201d3902921117776313b6fce098f7a2e3a8d8a8f02fc63fc0"} Oct 31 23:25:13.991519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:13.991494 5043 scope.go:122] "RemoveContainer" containerID="62774c08ba9a484dd0ebc15316de6103802c19591ca8999ef4eeadc376445b46" Oct 31 23:25:14.005701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.005673 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb"] Oct 31 23:25:14.010256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.010236 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-5ljbb"] Oct 31 23:25:14.022110 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.022087 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7"] Oct 31 23:25:14.030569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.030529 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-135248-openshift-default-5547d9c9f-2cwh7"] Oct 31 23:25:14.270623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.270581 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:14.640253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.640215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:14.911369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.909399 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:25:14.924580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.912183 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984437 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984493 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984531 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984580 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984621 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984666 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984718 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984760 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984811 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984841 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984880 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984921 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984972 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.984999 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.985030 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.985113 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.985176 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data\") pod \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\" (UID: \"be9e7b2a-02ec-4c6d-bea4-6dc49798d296\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.985205 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert\") pod \"cd24c44a-38fc-460e-a833-fd5308961380\" (UID: \"cd24c44a-38fc-460e-a833-fd5308961380\") " Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.994929 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.995980 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "kube-api-access-689g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.996837 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.997064 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.998008 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.998223 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.998502 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.999607 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.999699 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:14.999920 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.000100 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.000318 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.001770 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "kube-api-access-zfc6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.003803 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.003884 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.003933 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token" pod "be9e7b2a-02ec-4c6d-bea4-6dc49798d296" (UID: "be9e7b2a-02ec-4c6d-bea4-6dc49798d296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:15.014615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.006510 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:15.091873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.060697 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo" pod "cd24c44a-38fc-460e-a833-fd5308961380" (UID: "cd24c44a-38fc-460e-a833-fd5308961380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145113 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfc6b\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-kube-api-access-zfc6b\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145163 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145180 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145196 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145212 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145233 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145273 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd24c44a-38fc-460e-a833-fd5308961380-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145290 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-689g8\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-kube-api-access-689g8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145311 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145326 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145340 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd24c44a-38fc-460e-a833-fd5308961380-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145355 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145374 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145389 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145404 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145419 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd24c44a-38fc-460e-a833-fd5308961380-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145440 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd24c44a-38fc-460e-a833-fd5308961380-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.157931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.145455 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9e7b2a-02ec-4c6d-bea4-6dc49798d296-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.310365 5043 generic.go:356] "Generic (PLEG): container finished" podID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerID="ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355" exitCode=0 Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.310577 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" event={"ID":"be9e7b2a-02ec-4c6d-bea4-6dc49798d296","Type":"ContainerDied","Data":"ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355"} Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.310675 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" event={"ID":"be9e7b2a-02ec-4c6d-bea4-6dc49798d296","Type":"ContainerDied","Data":"e1ce472123a81e50b4a4ce1a51e9ff4c503486813e112eb37f904f172ad8fc11"} Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.310708 5043 scope.go:122] "RemoveContainer" containerID="ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355" Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.310868 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr" Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.335202 5043 generic.go:356] "Generic (PLEG): container finished" podID="cd24c44a-38fc-460e-a833-fd5308961380" containerID="c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d" exitCode=0 Oct 31 23:25:15.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.335481 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" Oct 31 23:25:15.344572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.336500 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" event={"ID":"cd24c44a-38fc-460e-a833-fd5308961380","Type":"ContainerDied","Data":"c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d"} Oct 31 23:25:15.344572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.336572 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf" event={"ID":"cd24c44a-38fc-460e-a833-fd5308961380","Type":"ContainerDied","Data":"b5a28576c1450b3f63f4daec8639da5cb0395d188fcbe0b3a41ff63c2509072e"} Oct 31 23:25:15.406894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.405110 5043 scope.go:122] "RemoveContainer" containerID="ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355" Oct 31 23:25:15.408200 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:15.407745 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355\": container with ID starting with ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355 not found: ID does not exist" containerID="ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355" Oct 31 23:25:15.408200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.407792 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355"} err="failed to get container status \"ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355\": rpc error: code = NotFound desc = could not find container \"ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355\": container with ID starting with ba160622c1f4df18cc9c2c15740189c0a9c2927add612a764f32dfb8dc4e5355 not found: ID does not exist" Oct 31 23:25:15.408200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.407819 5043 scope.go:122] "RemoveContainer" containerID="c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d" Oct 31 23:25:15.453599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.450721 5043 scope.go:122] "RemoveContainer" containerID="c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d" Oct 31 23:25:15.456540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.456506 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf"] Oct 31 23:25:15.466147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.466110 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-8pbtf"] Oct 31 23:25:15.494977 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:15.494941 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d\": container with ID starting with c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d not found: ID does not exist" containerID="c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d" Oct 31 23:25:15.495335 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.495003 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d"} err="failed to get container status \"c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d\": rpc error: code = NotFound desc = could not find container \"c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d\": container with ID starting with c9d6aabb59320d5a85d380bdce085c3004d8a2b1c2e1457fed6049d7bedbfc8d not found: ID does not exist" Oct 31 23:25:15.497222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.497200 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr"] Oct 31 23:25:15.502094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.502073 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7d7eed-openshift-default-54b5587f45-lwvtr"] Oct 31 23:25:15.648530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.648447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:15.935796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.935721 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" path="/var/lib/kubelet/pods/be9e7b2a-02ec-4c6d-bea4-6dc49798d296/volumes" Oct 31 23:25:15.936263 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.936244 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd24c44a-38fc-460e-a833-fd5308961380" path="/var/lib/kubelet/pods/cd24c44a-38fc-460e-a833-fd5308961380/volumes" Oct 31 23:25:15.936693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.936679 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" path="/var/lib/kubelet/pods/d336aa46-fc0a-4c34-a9dc-a45949939314/volumes" Oct 31 23:25:15.937479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:15.937462 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" path="/var/lib/kubelet/pods/f5fdaed9-d6a5-4748-b93a-d385a615164a/volumes" Oct 31 23:25:16.043994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:16.043958 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds"] Oct 31 23:25:16.051370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:16.051340 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp"] Oct 31 23:25:16.153216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:16.153152 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:16.295506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:16.295470 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:16.437189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:16.437139 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:17.108959 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.108921 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:17.240188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.240146 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:17.379272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.379187 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl"] Oct 31 23:25:17.379448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.379390 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" containerID="cri-o://7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f" gracePeriod=30 Oct 31 23:25:17.384752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.384716 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg"] Oct 31 23:25:17.384921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.384895 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" containerID="cri-o://e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0" gracePeriod=30 Oct 31 23:25:17.433146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.433106 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:17.813986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.813949 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:17.959955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:17.959916 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:18.061366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.061332 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn"] Oct 31 23:25:18.064662 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.064612 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e90c-xlljn"] Oct 31 23:25:18.084464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.084437 5043 patch_prober.go:29] interesting pod/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:18.084600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.084484 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:18.106189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.106160 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:18.253791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.253762 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:18.576033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:18.575992 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:19.029131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:19.029069 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:19.740501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:19.740421 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:19.953818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:19.953279 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8d6975-98da-402d-b929-b08fbf891ff6" path="/var/lib/kubelet/pods/5b8d6975-98da-402d-b929-b08fbf891ff6/volumes" Oct 31 23:25:20.243873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:20.243711 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs"] Oct 31 23:25:20.244112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:20.243983 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" containerID="cri-o://cb29e637a0267be632d53370dd52f10f13c4ef66db6297b1ce425f61d4c631d6" gracePeriod=30 Oct 31 23:25:20.248692 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:20.248646 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p"] Oct 31 23:25:20.248865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:20.248844 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" containerID="cri-o://b814ef7c3ca182666c0e2e8b2632d1ef0cfbdbba1c04f65825d7c0d0dad25522" gracePeriod=30 Oct 31 23:25:20.651284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:20.651196 5043 patch_prober.go:29] interesting pod/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:25:20.651284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:20.651269 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:25:21.044821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.044761 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" containerID="cri-o://b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01" gracePeriod=25 Oct 31 23:25:21.052362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.052326 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" containerID="cri-o://acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d" gracePeriod=25 Oct 31 23:25:21.356339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.356004 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:25:21.358028 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.357995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:21.391477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.391444 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:25:21.434494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.434459 5043 generic.go:356] "Generic (PLEG): container finished" podID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerID="b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01" exitCode=0 Oct 31 23:25:21.434713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.434554 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" event={"ID":"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1","Type":"ContainerDied","Data":"b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01"} Oct 31 23:25:21.434713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.434618 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" event={"ID":"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1","Type":"ContainerDied","Data":"009124b0e480b77a4101338c7eeb6488e0ed9e2adcd6f1d9ff1deb591bcaf950"} Oct 31 23:25:21.434713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.434648 5043 scope.go:122] "RemoveContainer" containerID="b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01" Oct 31 23:25:21.434922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.434785 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds" Oct 31 23:25:21.437374 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.437348 5043 generic.go:356] "Generic (PLEG): container finished" podID="3bf431e0-53a2-4422-905b-ae83975f1052" containerID="acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d" exitCode=0 Oct 31 23:25:21.437505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.437488 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" event={"ID":"3bf431e0-53a2-4422-905b-ae83975f1052","Type":"ContainerDied","Data":"acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d"} Oct 31 23:25:21.437610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.437529 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" event={"ID":"3bf431e0-53a2-4422-905b-ae83975f1052","Type":"ContainerDied","Data":"bcfd963bd6da4582bb533534f1bae554e8476d6ba8858c2e6161ef8560b37d2b"} Oct 31 23:25:21.437715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.437697 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp" Oct 31 23:25:21.448240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.448219 5043 scope.go:122] "RemoveContainer" containerID="b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01" Oct 31 23:25:21.448478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.448451 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume\") pod \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " Oct 31 23:25:21.448615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.448492 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx\" (UniqueName: \"kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx\") pod \"3bf431e0-53a2-4422-905b-ae83975f1052\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " Oct 31 23:25:21.448729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.448710 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume\") pod \"3bf431e0-53a2-4422-905b-ae83975f1052\" (UID: \"3bf431e0-53a2-4422-905b-ae83975f1052\") " Oct 31 23:25:21.448802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.448787 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6\" (UniqueName: \"kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6\") pod \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\" (UID: \"be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1\") " Oct 31 23:25:21.449360 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:21.449330 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01\": container with ID starting with b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01 not found: ID does not exist" containerID="b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01" Oct 31 23:25:21.449435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.449373 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01"} err="failed to get container status \"b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01\": rpc error: code = NotFound desc = could not find container \"b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01\": container with ID starting with b02c458bfb35d1c3f3aba1f104c6bf33f184f1edc2314de30cebfeff035e6b01 not found: ID does not exist" Oct 31 23:25:21.449435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.449395 5043 scope.go:122] "RemoveContainer" containerID="acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d" Oct 31 23:25:21.455498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.455470 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume" pod "be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" (UID: "be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:21.455606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.455507 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume" pod "3bf431e0-53a2-4422-905b-ae83975f1052" (UID: "3bf431e0-53a2-4422-905b-ae83975f1052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:21.457427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.457404 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx" pod "3bf431e0-53a2-4422-905b-ae83975f1052" (UID: "3bf431e0-53a2-4422-905b-ae83975f1052"). InnerVolumeSpecName "kube-api-access-rb8zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:21.457795 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.457763 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6" pod "be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" (UID: "be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1"). InnerVolumeSpecName "kube-api-access-26lh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:21.466934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.466914 5043 scope.go:122] "RemoveContainer" containerID="acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d" Oct 31 23:25:21.467222 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:21.467202 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d\": container with ID starting with acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d not found: ID does not exist" containerID="acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d" Oct 31 23:25:21.467297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.467231 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d"} err="failed to get container status \"acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d\": rpc error: code = NotFound desc = could not find container \"acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d\": container with ID starting with acf33f3e7e10c6361cb5e898167dc7411a0a8c9f68290537c5a45d3c9ed8344d not found: ID does not exist" Oct 31 23:25:21.550041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.549965 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lh6\" (UniqueName: \"kubernetes.io/projected/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-kube-api-access-26lh6\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:21.550041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.550003 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:21.550041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.550019 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8zx\" (UniqueName: \"kubernetes.io/projected/3bf431e0-53a2-4422-905b-ae83975f1052-kube-api-access-rb8zx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:21.550041 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.550034 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf431e0-53a2-4422-905b-ae83975f1052-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:21.766988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.766957 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp"] Oct 31 23:25:21.772864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.772841 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e90c-c46689d9f-rk2tp"] Oct 31 23:25:21.784698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.784673 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds"] Oct 31 23:25:21.789670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.789647 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e90c-8597c64598-xj4ds"] Oct 31 23:25:21.926867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.926781 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:21.936446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.936410 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" path="/var/lib/kubelet/pods/3bf431e0-53a2-4422-905b-ae83975f1052/volumes" Oct 31 23:25:21.936940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:21.936918 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" path="/var/lib/kubelet/pods/be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1/volumes" Oct 31 23:25:22.639270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.633086 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:22.704582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.704516 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:25:22.708641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.708621 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:25:22.785250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785217 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785270 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.785423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785293 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785315 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785348 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785376 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.785423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785405 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785441 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785475 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785529 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785624 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785660 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785675 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785692 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785687 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.785800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785775 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785823 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data\") pod \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\" (UID: \"d0b6666c-4eae-49da-9e9b-d2806ae3ed50\") " Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785852 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785763 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785887 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785889 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.785952 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8\") pod \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\" (UID: \"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f\") " Oct 31 23:25:22.786293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.786095 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.786962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.786796 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.786962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.786907 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.787105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.786986 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.787105 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787004 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787233 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787261 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787277 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787293 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787308 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787323 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787337 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787340 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787352 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.787391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.787365 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.789747 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.789715 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "kube-api-access-vtfrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.789951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.789922 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.789951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.789925 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:22.790133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.790072 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.790133 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.790123 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.790431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.790399 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy" pod "d0b6666c-4eae-49da-9e9b-d2806ae3ed50" (UID: "d0b6666c-4eae-49da-9e9b-d2806ae3ed50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:22.790523 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.790475 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:22.791366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.791339 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8" pod "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" (UID: "e8e8a9f3-8353-41c3-9ba1-d1adf0de151f"). InnerVolumeSpecName "kube-api-access-nntw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:22.805444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.805410 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:22.888712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888667 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888704 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtfrj\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-kube-api-access-vtfrj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888720 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888733 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888745 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888768 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888779 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b6666c-4eae-49da-9e9b-d2806ae3ed50-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888791 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:22.888998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:22.888803 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntw8\" (UniqueName: \"kubernetes.io/projected/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f-kube-api-access-nntw8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:23.457308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.457221 5043 generic.go:356] "Generic (PLEG): container finished" podID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerID="e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0" exitCode=0 Oct 31 23:25:23.457308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.457274 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" Oct 31 23:25:23.457308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.457298 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" event={"ID":"d0b6666c-4eae-49da-9e9b-d2806ae3ed50","Type":"ContainerDied","Data":"e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0"} Oct 31 23:25:23.457604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.457351 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg" event={"ID":"d0b6666c-4eae-49da-9e9b-d2806ae3ed50","Type":"ContainerDied","Data":"ef85cf096a215b263ff479c987c37626d3f9f2fb09c84b8a31886b1609c1913a"} Oct 31 23:25:23.457604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.457378 5043 scope.go:122] "RemoveContainer" containerID="e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0" Oct 31 23:25:23.458846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.458823 5043 generic.go:356] "Generic (PLEG): container finished" podID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerID="7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f" exitCode=0 Oct 31 23:25:23.458993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.458876 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" Oct 31 23:25:23.458993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.458874 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" event={"ID":"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f","Type":"ContainerDied","Data":"7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f"} Oct 31 23:25:23.459112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.458999 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl" event={"ID":"e8e8a9f3-8353-41c3-9ba1-d1adf0de151f","Type":"ContainerDied","Data":"4c2f27748403961546865d8fbc0a4cd20b7cacfafa108e250e851826808704bf"} Oct 31 23:25:23.470922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.470906 5043 scope.go:122] "RemoveContainer" containerID="e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0" Oct 31 23:25:23.471196 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:23.471176 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0\": container with ID starting with e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0 not found: ID does not exist" containerID="e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0" Oct 31 23:25:23.471269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.471208 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0"} err="failed to get container status \"e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0\": rpc error: code = NotFound desc = could not find container \"e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0\": container with ID starting with e2303e201cc4c14a0e395b52c5bc39c50ee00f670b9a1dbb98ed9804d49bb8a0 not found: ID does not exist" Oct 31 23:25:23.471269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.471233 5043 scope.go:122] "RemoveContainer" containerID="7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f" Oct 31 23:25:23.482108 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.482088 5043 scope.go:122] "RemoveContainer" containerID="7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f" Oct 31 23:25:23.482386 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:23.482364 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f\": container with ID starting with 7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f not found: ID does not exist" containerID="7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f" Oct 31 23:25:23.482455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.482392 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f"} err="failed to get container status \"7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f\": rpc error: code = NotFound desc = could not find container \"7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f\": container with ID starting with 7c173d74726da9584e0fe63cdd90e9de222469172f70123dc5f33ce67342715f not found: ID does not exist" Oct 31 23:25:23.488944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.488920 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg"] Oct 31 23:25:23.493901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.493881 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-ml6gg"] Oct 31 23:25:23.508051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.508029 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl"] Oct 31 23:25:23.514037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.514016 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-67fce9-openshift-default-647f9cddd7-wwknl"] Oct 31 23:25:23.604497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.604468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:23.936297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.936255 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" path="/var/lib/kubelet/pods/d0b6666c-4eae-49da-9e9b-d2806ae3ed50/volumes" Oct 31 23:25:23.936926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:23.936903 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" path="/var/lib/kubelet/pods/e8e8a9f3-8353-41c3-9ba1-d1adf0de151f/volumes" Oct 31 23:25:24.438036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:24.436686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:25.173716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.173679 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:25.314015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.313976 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:25.483217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.483130 5043 generic.go:356] "Generic (PLEG): container finished" podID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerID="cb29e637a0267be632d53370dd52f10f13c4ef66db6297b1ce425f61d4c631d6" exitCode=0 Oct 31 23:25:25.483217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.483199 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" event={"ID":"aa6404e0-d1f4-48c9-a7be-631d72442e4e","Type":"ContainerDied","Data":"cb29e637a0267be632d53370dd52f10f13c4ef66db6297b1ce425f61d4c631d6"} Oct 31 23:25:25.485575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.485536 5043 generic.go:356] "Generic (PLEG): container finished" podID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerID="b814ef7c3ca182666c0e2e8b2632d1ef0cfbdbba1c04f65825d7c0d0dad25522" exitCode=0 Oct 31 23:25:25.485708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.485579 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" event={"ID":"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2","Type":"ContainerDied","Data":"b814ef7c3ca182666c0e2e8b2632d1ef0cfbdbba1c04f65825d7c0d0dad25522"} Oct 31 23:25:25.485708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.485648 5043 scope.go:122] "RemoveContainer" containerID="ce420c0a0725ea9b19750cbeca5f764501dee13c2467b916c8bee5365e7fa2cb" Oct 31 23:25:25.549852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.549822 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:25:25.556616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.556185 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:25.645435 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.645391 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:25:25.728553 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728507 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728576 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728617 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728669 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.728719 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728707 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728744 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728772 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728814 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728861 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728906 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728931 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.728984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728965 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728980 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.728993 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729049 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729089 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729130 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729165 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket\") pod \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\" (UID: \"aa6404e0-d1f4-48c9-a7be-631d72442e4e\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729215 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729249 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n\") pod \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\" (UID: \"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2\") " Oct 31 23:25:25.729361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729256 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.729764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729462 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:25.729764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729738 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.729848 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729829 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.729885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729851 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.729885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729870 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.729948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729885 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.729948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729899 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.730012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.729940 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.730782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.730375 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:25.730782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.730421 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.730782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.730436 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.730782 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.730650 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.731823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.731799 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:25.732244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.732218 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.732947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.732918 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.733720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.733664 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.733814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.733793 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n" pod "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" (UID: "084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2"). InnerVolumeSpecName "kube-api-access-4h68n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.733863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.733806 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:25:25.733947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.733933 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "kube-api-access-zjvgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:25.734189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.734169 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy" pod "aa6404e0-d1f4-48c9-a7be-631d72442e4e" (UID: "aa6404e0-d1f4-48c9-a7be-631d72442e4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:25:25.831291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831261 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831290 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h68n\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-kube-api-access-4h68n\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831301 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831315 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831328 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjvgf\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-kube-api-access-zjvgf\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831338 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831347 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831355 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831364 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831373 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831382 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831390 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6404e0-d1f4-48c9-a7be-631d72442e4e-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:25.831495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:25.831398 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6404e0-d1f4-48c9-a7be-631d72442e4e-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:26.006369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.006333 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:26.144100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.144046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:26.169013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.168978 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz"] Oct 31 23:25:26.169440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169426 5043 cpu_manager.go:414] "containerMap: removing container" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169441 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169453 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169459 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169465 5043 cpu_manager.go:414] "containerMap: removing container" podUID="5b8d6975-98da-402d-b929-b08fbf891ff6" containerName="revoke-keys" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169471 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8d6975-98da-402d-b929-b08fbf891ff6" containerName="revoke-keys" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169479 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169484 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" Oct 31 23:25:26.169490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169492 5043 cpu_manager.go:414] "containerMap: removing container" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169497 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169504 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169508 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169517 5043 cpu_manager.go:414] "containerMap: removing container" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169521 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169528 5043 cpu_manager.go:414] "containerMap: removing container" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169532 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169539 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169561 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169571 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169576 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169591 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169596 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169604 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169608 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169623 5043 cpu_manager.go:414] "containerMap: removing container" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169628 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169635 5043 cpu_manager.go:414] "containerMap: removing container" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169639 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169649 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169653 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169714 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4c1dc7-8374-4d2e-bedf-3a7a39abdfd1" containerName="payload-processing" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169724 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169731 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e8a9f3-8353-41c3-9ba1-d1adf0de151f" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169738 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169745 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9e7b2a-02ec-4c6d-bea4-6dc49798d296" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169750 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8d6975-98da-402d-b929-b08fbf891ff6" containerName="revoke-keys" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169755 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b6666c-4eae-49da-9e9b-d2806ae3ed50" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169760 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="01442497-4a44-42db-bd0d-7d7839a7efe1" containerName="main" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169764 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fdaed9-d6a5-4748-b93a-d385a615164a" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169770 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d336aa46-fc0a-4c34-a9dc-a45949939314" containerName="istio-proxy" Oct 31 23:25:26.169761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169776 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd24c44a-38fc-460e-a833-fd5308961380" containerName="istio-proxy" Oct 31 23:25:26.170846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169781 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" Oct 31 23:25:26.170846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169786 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf431e0-53a2-4422-905b-ae83975f1052" containerName="payload-pre-processing" Oct 31 23:25:26.170846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169792 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d9c0df-ff9e-4bab-8621-ee2537b266de" containerName="istio-proxy" Oct 31 23:25:26.170846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169797 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f8db39-ca3d-4e82-a6de-774b40cb6325" containerName="main" Oct 31 23:25:26.170846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.169805 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" containerName="istio-proxy" Oct 31 23:25:26.174503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.174485 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.178817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.178792 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:25:26.178817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.178810 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:25:26.181042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.180997 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz"] Oct 31 23:25:26.265192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.265100 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:25:26.265344 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.265217 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:25:26.346732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.346690 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.346893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.346753 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.346893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.346844 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5sw\" (UniqueName: \"kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.447598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.447528 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.447753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.447697 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5sw\" (UniqueName: \"kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.447858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.447846 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.448253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.448222 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.450817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.450791 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.465455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.465418 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5sw\" (UniqueName: \"kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw\") pod \"maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.497348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.497317 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" Oct 31 23:25:26.497348 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.497328 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs" event={"ID":"aa6404e0-d1f4-48c9-a7be-631d72442e4e","Type":"ContainerDied","Data":"a43b14a9f9e0c61c9619057bdb63afdcfff4753dd89a8cfd6448b6bb7752773e"} Oct 31 23:25:26.497607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.497445 5043 scope.go:122] "RemoveContainer" containerID="cb29e637a0267be632d53370dd52f10f13c4ef66db6297b1ce425f61d4c631d6" Oct 31 23:25:26.500709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.500674 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" event={"ID":"084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2","Type":"ContainerDied","Data":"c4ca6bc0d850fc04bbcba1a196243579fc90d5c58ef7776e71aabe90b7ebee9e"} Oct 31 23:25:26.500816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.500710 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p" Oct 31 23:25:26.513205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.513185 5043 scope.go:122] "RemoveContainer" containerID="b814ef7c3ca182666c0e2e8b2632d1ef0cfbdbba1c04f65825d7c0d0dad25522" Oct 31 23:25:26.526485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.526438 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs"] Oct 31 23:25:26.531846 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.531824 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-dz7vs"] Oct 31 23:25:26.537075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.537049 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:26.554476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.554443 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p"] Oct 31 23:25:26.558336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.558306 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04c5c2-openshift-default-6669c5889-vwr9p"] Oct 31 23:25:26.569610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.569574 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:26.698869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.698817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:26.846656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:26.846625 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz"] Oct 31 23:25:26.848521 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:25:26.848490 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7f125b_21be_422e_a4ca_ac7a4d88b61f.slice/crio-6f08d53a2eb3cb9a3b82b77f137b25469b16c5dc93d0040db95fde68758ce867 WatchSource:0}: Error finding container 6f08d53a2eb3cb9a3b82b77f137b25469b16c5dc93d0040db95fde68758ce867: Status 404 returned error can't find the container with id 6f08d53a2eb3cb9a3b82b77f137b25469b16c5dc93d0040db95fde68758ce867 Oct 31 23:25:27.098178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.098083 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:27.511486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.511444 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" event={"ID":"bf7f125b-21be-422e-a4ca-ac7a4d88b61f","Type":"ContainerStarted","Data":"6011a905bc22fdc4078eb0585a55febbbf6ced3c8e3741c46db24279084bccb8"} Oct 31 23:25:27.511978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.511509 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" event={"ID":"bf7f125b-21be-422e-a4ca-ac7a4d88b61f","Type":"ContainerStarted","Data":"6f08d53a2eb3cb9a3b82b77f137b25469b16c5dc93d0040db95fde68758ce867"} Oct 31 23:25:27.531694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.531631 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" podStartSLOduration=1.531613473 podStartE2EDuration="1.531613473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:25:27.530171552 +0000 UTC m=+2153.759725964" watchObservedRunningTime="2026-10-31 23:25:27.531613473 +0000 UTC m=+2153.761167884" Oct 31 23:25:27.803419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.803284 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:27.936590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.936536 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" path="/var/lib/kubelet/pods/084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2/volumes" Oct 31 23:25:27.937053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:27.937040 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6404e0-d1f4-48c9-a7be-631d72442e4e" path="/var/lib/kubelet/pods/aa6404e0-d1f4-48c9-a7be-631d72442e4e/volumes" Oct 31 23:25:28.134894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:28.134785 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:28.523670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:28.523632 5043 generic.go:356] "Generic (PLEG): container finished" podID="bf7f125b-21be-422e-a4ca-ac7a4d88b61f" containerID="6011a905bc22fdc4078eb0585a55febbbf6ced3c8e3741c46db24279084bccb8" exitCode=0 Oct 31 23:25:28.524157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:28.523687 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" event={"ID":"bf7f125b-21be-422e-a4ca-ac7a4d88b61f","Type":"ContainerDied","Data":"6011a905bc22fdc4078eb0585a55febbbf6ced3c8e3741c46db24279084bccb8"} Oct 31 23:25:28.718922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:28.718886 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:29.456898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.456850 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:29.699826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.699796 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:29.782885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.782824 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca\") pod \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " Oct 31 23:25:29.783087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.782960 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca\") pod \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " Oct 31 23:25:29.783087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.783047 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw\" (UniqueName: \"kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw\") pod \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\" (UID: \"bf7f125b-21be-422e-a4ca-ac7a4d88b61f\") " Oct 31 23:25:29.783331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.783303 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca" pod "bf7f125b-21be-422e-a4ca-ac7a4d88b61f" (UID: "bf7f125b-21be-422e-a4ca-ac7a4d88b61f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:29.783529 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.783504 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:29.785245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.785214 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca" pod "bf7f125b-21be-422e-a4ca-ac7a4d88b61f" (UID: "bf7f125b-21be-422e-a4ca-ac7a4d88b61f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:29.785364 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.785341 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw" pod "bf7f125b-21be-422e-a4ca-ac7a4d88b61f" (UID: "bf7f125b-21be-422e-a4ca-ac7a4d88b61f"). InnerVolumeSpecName "kube-api-access-dt5sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:29.884249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.884200 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5sw\" (UniqueName: \"kubernetes.io/projected/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-kube-api-access-dt5sw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:29.884249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:29.884246 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7f125b-21be-422e-a4ca-ac7a4d88b61f-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:30.219192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:30.218920 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:30.545040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:30.545013 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" Oct 31 23:25:30.545339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:30.545006 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz" event={"ID":"bf7f125b-21be-422e-a4ca-ac7a4d88b61f","Type":"ContainerDied","Data":"6f08d53a2eb3cb9a3b82b77f137b25469b16c5dc93d0040db95fde68758ce867"} Oct 31 23:25:30.545339 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:30.545144 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f08d53a2eb3cb9a3b82b77f137b25469b16c5dc93d0040db95fde68758ce867" Oct 31 23:25:30.813574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:30.813445 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:31.128971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:31.128866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:31.497061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:31.497013 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:32.198451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:32.198372 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:32.837420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:32.837332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:33.506036 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:33.506002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:33.644375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:33.644334 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:33.782507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:33.781326 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:34.507142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:34.507095 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:34.840352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:34.840265 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:35.304227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:35.304188 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:35.444141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:35.444097 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:36.144299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:36.144256 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:36.366685 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:36.366639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:36.510967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:36.510929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:36.722580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:36.722493 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:37.152433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152395 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr"] Oct 31 23:25:37.152930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152915 5043 cpu_manager.go:414] "containerMap: removing container" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" Oct 31 23:25:37.152993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152932 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084707a6-e9ab-4b8f-9f76-9f3a8f7d53c2" containerName="istio-proxy" Oct 31 23:25:37.152993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152948 5043 cpu_manager.go:414] "containerMap: removing container" podUID="bf7f125b-21be-422e-a4ca-ac7a4d88b61f" containerName="revoke-keys" Oct 31 23:25:37.152993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152953 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7f125b-21be-422e-a4ca-ac7a4d88b61f" containerName="revoke-keys" Oct 31 23:25:37.152993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152965 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" Oct 31 23:25:37.152993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.152971 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" Oct 31 23:25:37.153170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.153060 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7f125b-21be-422e-a4ca-ac7a4d88b61f" containerName="revoke-keys" Oct 31 23:25:37.153170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.153071 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef7adc6-4f5e-4875-84bc-629aeb4d2eef" containerName="istio-proxy" Oct 31 23:25:37.157334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.157307 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg"] Oct 31 23:25:37.157493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.157427 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.162998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.162969 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.164039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.164010 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:25:37.164736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.164712 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:25:37.164929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.164804 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:25:37.164929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.164846 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:25:37.168436 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.168416 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr"] Oct 31 23:25:37.172359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.172329 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg"] Oct 31 23:25:37.255773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.255735 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhs9\" (UniqueName: \"kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.255962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.255862 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.256011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.255958 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2dx\" (UniqueName: \"kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.256011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.256002 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.256086 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.256024 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.256145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.256124 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.357469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.357424 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.357710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.357579 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2dx\" (UniqueName: \"kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.357710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.357614 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.357710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.357638 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.357710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.357675 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.357952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.357735 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhs9\" (UniqueName: \"kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.358438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.358389 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.358625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.358585 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.360710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.360687 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.360843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.360720 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.372803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.372766 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2dx\" (UniqueName: \"kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx\") pod \"maas-api-revoke-sub-022de01f-lw5bg\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.372989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.372836 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhs9\" (UniqueName: \"kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9\") pod \"maas-api-revoke-sub-1666a970-97jzr\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.485707 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.485608 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:37.491122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.491099 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:37.780447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.780412 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr"] Oct 31 23:25:37.781908 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:25:37.781881 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74453611_c34e_450c_ad24_bb80abbeaf01.slice/crio-35988e502df2c32e912b159a1ca07bf18ee3d30efd40e7c8fa4d2d160bbdaaf9 WatchSource:0}: Error finding container 35988e502df2c32e912b159a1ca07bf18ee3d30efd40e7c8fa4d2d160bbdaaf9: Status 404 returned error can't find the container with id 35988e502df2c32e912b159a1ca07bf18ee3d30efd40e7c8fa4d2d160bbdaaf9 Oct 31 23:25:37.794675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:37.794644 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg"] Oct 31 23:25:37.797167 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:25:37.797136 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42ff2ef_0147_4c3f_a0f3_0cb0b1ba4d41.slice/crio-dcc6d314bf7ea6fa3bfe363b1aa94bfbadabdff70a6b2cabdf4269a799478046 WatchSource:0}: Error finding container dcc6d314bf7ea6fa3bfe363b1aa94bfbadabdff70a6b2cabdf4269a799478046: Status 404 returned error can't find the container with id dcc6d314bf7ea6fa3bfe363b1aa94bfbadabdff70a6b2cabdf4269a799478046 Oct 31 23:25:38.609118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:38.609078 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" event={"ID":"74453611-c34e-450c-ad24-bb80abbeaf01","Type":"ContainerStarted","Data":"d9b869063e60fc567125c87ae1369e33b5ab3bdfc94998891c68996edc3a7248"} Oct 31 23:25:38.609646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:38.609145 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" event={"ID":"74453611-c34e-450c-ad24-bb80abbeaf01","Type":"ContainerStarted","Data":"35988e502df2c32e912b159a1ca07bf18ee3d30efd40e7c8fa4d2d160bbdaaf9"} Oct 31 23:25:38.611080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:38.611047 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" event={"ID":"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41","Type":"ContainerStarted","Data":"c6750ad76a463853e92597d594e132074dce1836e71fa5b263a079f6d7cd0c67"} Oct 31 23:25:38.611218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:38.611103 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" event={"ID":"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41","Type":"ContainerStarted","Data":"dcc6d314bf7ea6fa3bfe363b1aa94bfbadabdff70a6b2cabdf4269a799478046"} Oct 31 23:25:38.630367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:38.630314 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" podStartSLOduration=1.6303013389999998 podStartE2EDuration="1.630301339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:25:38.627989988 +0000 UTC m=+2164.857544399" watchObservedRunningTime="2026-10-31 23:25:38.630301339 +0000 UTC m=+2164.859855753" Oct 31 23:25:38.647358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:38.647298 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" podStartSLOduration=1.647283131 podStartE2EDuration="1.647283131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:25:38.647071259 +0000 UTC m=+2164.876625671" watchObservedRunningTime="2026-10-31 23:25:38.647283131 +0000 UTC m=+2164.876837541" Oct 31 23:25:39.623661 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:39.623617 5043 generic.go:356] "Generic (PLEG): container finished" podID="74453611-c34e-450c-ad24-bb80abbeaf01" containerID="d9b869063e60fc567125c87ae1369e33b5ab3bdfc94998891c68996edc3a7248" exitCode=0 Oct 31 23:25:39.624179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:39.623714 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" event={"ID":"74453611-c34e-450c-ad24-bb80abbeaf01","Type":"ContainerDied","Data":"d9b869063e60fc567125c87ae1369e33b5ab3bdfc94998891c68996edc3a7248"} Oct 31 23:25:39.625628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:39.625599 5043 generic.go:356] "Generic (PLEG): container finished" podID="b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" containerID="c6750ad76a463853e92597d594e132074dce1836e71fa5b263a079f6d7cd0c67" exitCode=0 Oct 31 23:25:39.625750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:39.625689 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" event={"ID":"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41","Type":"ContainerDied","Data":"c6750ad76a463853e92597d594e132074dce1836e71fa5b263a079f6d7cd0c67"} Oct 31 23:25:40.367786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.367753 5043 scope.go:122] "RemoveContainer" containerID="23fca3cc4cc839371091a211e909edffc0770eba7d2c7d79b9d1d02b695df0eb" Oct 31 23:25:40.461936 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:25:40.461863 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:25:40.814930 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.814898 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:40.877357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.877322 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:40.905299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.905220 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca\") pod \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " Oct 31 23:25:40.905469 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.905452 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx\" (UniqueName: \"kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx\") pod \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " Oct 31 23:25:40.905531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.905485 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca\") pod \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\" (UID: \"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41\") " Oct 31 23:25:40.905943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.905916 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca" pod "b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" (UID: "b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:40.907696 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.907665 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca" pod "b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" (UID: "b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:40.907826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:40.907798 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx" pod "b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" (UID: "b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41"). InnerVolumeSpecName "kube-api-access-xr2dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:41.006956 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.006909 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9\" (UniqueName: \"kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9\") pod \"74453611-c34e-450c-ad24-bb80abbeaf01\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " Oct 31 23:25:41.007118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007062 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca\") pod \"74453611-c34e-450c-ad24-bb80abbeaf01\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " Oct 31 23:25:41.007118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007090 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca\") pod \"74453611-c34e-450c-ad24-bb80abbeaf01\" (UID: \"74453611-c34e-450c-ad24-bb80abbeaf01\") " Oct 31 23:25:41.007492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007465 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca" pod "74453611-c34e-450c-ad24-bb80abbeaf01" (UID: "74453611-c34e-450c-ad24-bb80abbeaf01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:25:41.007712 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007688 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:41.007780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007719 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2dx\" (UniqueName: \"kubernetes.io/projected/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-kube-api-access-xr2dx\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:41.007780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007735 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:41.007780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.007751 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:41.009227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.009206 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca" pod "74453611-c34e-450c-ad24-bb80abbeaf01" (UID: "74453611-c34e-450c-ad24-bb80abbeaf01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:25:41.009227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.009215 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9" pod "74453611-c34e-450c-ad24-bb80abbeaf01" (UID: "74453611-c34e-450c-ad24-bb80abbeaf01"). InnerVolumeSpecName "kube-api-access-9xhs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:25:41.109156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.109115 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74453611-c34e-450c-ad24-bb80abbeaf01-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:41.109156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.109150 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhs9\" (UniqueName: \"kubernetes.io/projected/74453611-c34e-450c-ad24-bb80abbeaf01-kube-api-access-9xhs9\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:25:41.360106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.360068 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:41.643210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.643114 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" Oct 31 23:25:41.643210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.643128 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr" event={"ID":"74453611-c34e-450c-ad24-bb80abbeaf01","Type":"ContainerDied","Data":"35988e502df2c32e912b159a1ca07bf18ee3d30efd40e7c8fa4d2d160bbdaaf9"} Oct 31 23:25:41.643210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.643182 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35988e502df2c32e912b159a1ca07bf18ee3d30efd40e7c8fa4d2d160bbdaaf9" Oct 31 23:25:41.644746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.644722 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" Oct 31 23:25:41.644905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.644757 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg" event={"ID":"b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41","Type":"ContainerDied","Data":"dcc6d314bf7ea6fa3bfe363b1aa94bfbadabdff70a6b2cabdf4269a799478046"} Oct 31 23:25:41.644905 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:41.644807 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc6d314bf7ea6fa3bfe363b1aa94bfbadabdff70a6b2cabdf4269a799478046" Oct 31 23:25:51.916465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:51.916426 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:52.078201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.078155 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:52.274921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.274888 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:52.365243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.365195 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz"] Oct 31 23:25:52.370146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.370110 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-255748-4k2bz"] Oct 31 23:25:52.417099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.417064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:52.549131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.549039 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:52.982132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:52.982033 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:53.935910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:53.935872 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7f125b-21be-422e-a4ca-ac7a4d88b61f" path="/var/lib/kubelet/pods/bf7f125b-21be-422e-a4ca-ac7a4d88b61f/volumes" Oct 31 23:25:59.161232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.161194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:59.301175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.301140 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:25:59.468445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.468273 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7"] Oct 31 23:25:59.468983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.468942 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" containerID="cri-o://230c54538a0e43eb31c7e3298d818d1625fed8ed4315d01d9f7b297581a5922a" gracePeriod=30 Oct 31 23:25:59.472775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.472749 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7"] Oct 31 23:25:59.473001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.472963 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" containerID="cri-o://cf7e673bb289d5670fabaa0b72aa93934c4ebf59e72e401a901ca53b45194996" gracePeriod=30 Oct 31 23:25:59.546365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:25:59.546327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:04.965426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:04.965329 5043 generic.go:356] "Generic (PLEG): container finished" podID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerID="cf7e673bb289d5670fabaa0b72aa93934c4ebf59e72e401a901ca53b45194996" exitCode=0 Oct 31 23:26:04.966229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:04.966199 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" event={"ID":"87090fa8-2197-43d2-8bcf-6429c94ef477","Type":"ContainerDied","Data":"cf7e673bb289d5670fabaa0b72aa93934c4ebf59e72e401a901ca53b45194996"} Oct 31 23:26:04.966434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:04.966417 5043 scope.go:122] "RemoveContainer" containerID="33081f0a5476a3fd7533ac250a7fca332f2b6f2a4a465a7d7c6e145ec761c02c" Oct 31 23:26:05.003025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.002989 5043 generic.go:356] "Generic (PLEG): container finished" podID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerID="230c54538a0e43eb31c7e3298d818d1625fed8ed4315d01d9f7b297581a5922a" exitCode=0 Oct 31 23:26:05.003323 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.003276 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" event={"ID":"dd1ae77f-51b8-4d9b-875b-daede18c8ab8","Type":"ContainerDied","Data":"230c54538a0e43eb31c7e3298d818d1625fed8ed4315d01d9f7b297581a5922a"} Oct 31 23:26:05.875126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.874748 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925093 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925166 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925220 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925261 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925307 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925376 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925425 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925536 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.925581 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token\") pod \"87090fa8-2197-43d2-8bcf-6429c94ef477\" (UID: \"87090fa8-2197-43d2-8bcf-6429c94ef477\") " Oct 31 23:26:05.938732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.930931 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:26:05.947128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.940665 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.959032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.954386 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.959032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.954651 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:05.959032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:05.957699 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.038597 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.015761 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.053256 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.053300 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.053315 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.053329 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.053344 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87090fa8-2197-43d2-8bcf-6429c94ef477-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.054662 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:26:06.058568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.054767 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "kube-api-access-qxcf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:06.099583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.078661 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.099583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.084725 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token" pod "87090fa8-2197-43d2-8bcf-6429c94ef477" (UID: "87090fa8-2197-43d2-8bcf-6429c94ef477"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:06.099583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.097439 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" event={"ID":"dd1ae77f-51b8-4d9b-875b-daede18c8ab8","Type":"ContainerDied","Data":"c681912c8bc40880e9330184c26363b0a6019fec0de6369056f568893b0ff11b"} Oct 31 23:26:06.099583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.097505 5043 scope.go:122] "RemoveContainer" containerID="230c54538a0e43eb31c7e3298d818d1625fed8ed4315d01d9f7b297581a5922a" Oct 31 23:26:06.099583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.097622 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7" Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161133 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161193 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161243 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161293 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161334 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161399 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161453 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161497 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161584 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs\") pod \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\" (UID: \"dd1ae77f-51b8-4d9b-875b-daede18c8ab8\") " Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161940 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcf2\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-kube-api-access-qxcf2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161961 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161978 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.175613 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.161992 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87090fa8-2197-43d2-8bcf-6429c94ef477-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.185585 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.185251 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:26:06.186167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.185835 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.189141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.188844 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.189141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.188981 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" event={"ID":"87090fa8-2197-43d2-8bcf-6429c94ef477","Type":"ContainerDied","Data":"8844c772dce494ff2580c5d3a8a32f57b9c8a4abe72863d268145684f684b819"} Oct 31 23:26:06.189141 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.189106 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7" Oct 31 23:26:06.192113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.191773 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:06.194720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.194359 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "kube-api-access-mw9v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:26:06.195140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.194890 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.195757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.195419 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:26:06.201912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.201886 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.215579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.212654 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket" pod "dd1ae77f-51b8-4d9b-875b-daede18c8ab8" (UID: "dd1ae77f-51b8-4d9b-875b-daede18c8ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:26:06.262594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.249341 5043 scope.go:122] "RemoveContainer" containerID="cf7e673bb289d5670fabaa0b72aa93934c4ebf59e72e401a901ca53b45194996" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264372 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264405 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264423 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264438 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264453 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264468 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw9v4\" (UniqueName: \"kubernetes.io/projected/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-kube-api-access-mw9v4\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264483 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264496 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.271498 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.264511 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd1ae77f-51b8-4d9b-875b-daede18c8ab8-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:26:06.324809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.324774 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7"] Oct 31 23:26:06.364493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.364398 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-497d7"] Oct 31 23:26:06.428605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.428564 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7"] Oct 31 23:26:06.433825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:06.433799 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-255748-openshift-default-59c49854d7-vr2m7"] Oct 31 23:26:07.892138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:07.890833 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:07.939571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:07.939512 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" path="/var/lib/kubelet/pods/87090fa8-2197-43d2-8bcf-6429c94ef477/volumes" Oct 31 23:26:07.940330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:07.940301 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" path="/var/lib/kubelet/pods/dd1ae77f-51b8-4d9b-875b-daede18c8ab8/volumes" Oct 31 23:26:08.079309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:08.079276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:09.286534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:09.286501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:10.459662 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:26:10.459616 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:26:14.657684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:14.657580 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:15.018722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:15.018680 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:17.621150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:17.621122 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:26:17.627459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:17.627431 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:26:17.638373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:17.638353 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:26:18.938520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:18.938493 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:26:18.943691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:18.943668 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:26:18.953637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:18.953614 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:26:33.025624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:33.025588 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46"] Oct 31 23:26:33.028746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:33.028715 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1512cc4-skk46"] Oct 31 23:26:33.655933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:33.655897 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:33.936184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:33.936100 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95df7923-e428-495b-a3a4-ea99cb1b013e" path="/var/lib/kubelet/pods/95df7923-e428-495b-a3a4-ea99cb1b013e/volumes" Oct 31 23:26:34.027447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:34.027409 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g"] Oct 31 23:26:34.032492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:34.032464 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79"] Oct 31 23:26:34.035608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:34.035580 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-255f7dcf-bl96g"] Oct 31 23:26:34.038123 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:34.038102 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ad7fa-wsj79"] Oct 31 23:26:35.559378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:35.559345 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:35.936220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:35.936137 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bd6f6f-3f7e-46d2-87e1-2d4437233318" path="/var/lib/kubelet/pods/15bd6f6f-3f7e-46d2-87e1-2d4437233318/volumes" Oct 31 23:26:35.936610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:35.936596 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="795bdc7a-da31-4a12-afad-cdc0ad0f7fed" path="/var/lib/kubelet/pods/795bdc7a-da31-4a12-afad-cdc0ad0f7fed/volumes" Oct 31 23:26:40.456183 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:26:40.456143 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:26:54.934810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:54.934769 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:26:55.076082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:26:55.076045 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:27:10.601863 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:27:10.601800 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:27:11.720700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:11.720647 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-x9wxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:27:11.721341 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:11.721301 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:27:11.836371 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:11.836328 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jqhtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:27:11.836593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:11.836410 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:27:11.899684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:11.899645 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-p9wjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:27:11.899856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:11.899728 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:27:14.726083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:14.726044 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:27:25.025733 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:25.025699 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4"] Oct 31 23:27:25.029122 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:25.029092 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9917efc9-qzpn4"] Oct 31 23:27:25.935760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:25.935729 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e494ca7-3dba-4df4-ad21-fcddadc8121d" path="/var/lib/kubelet/pods/0e494ca7-3dba-4df4-ad21-fcddadc8121d/volumes" Oct 31 23:27:40.455331 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:27:40.455287 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:27:40.602203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:40.602171 5043 scope.go:122] "RemoveContainer" containerID="d7bb61de56151f81646839a04268d7dd53541c313e258365d103014c42ad5e72" Oct 31 23:27:40.612270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:40.612243 5043 scope.go:122] "RemoveContainer" containerID="5c73ad771236e3c3375ef292a802d5c1364e3dd14277c0f5525afe2a22c6c991" Oct 31 23:27:40.621808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:40.621789 5043 scope.go:122] "RemoveContainer" containerID="8a1de1d51c2a93377f715c869cebbc87cc331d00a26a81d5495d7c5305d2f15b" Oct 31 23:27:40.629933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:40.629910 5043 scope.go:122] "RemoveContainer" containerID="2758fbd6565169c3d9119226ad06d6bb67c0bf20bccb547ef5cc994e1cae9c60" Oct 31 23:27:46.160216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:46.160137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:27:46.301955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:27:46.301917 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:02.514431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:02.514396 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:02.654559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:02.654512 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:05.354492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:05.354460 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:05.497975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:05.497939 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:10.457168 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:28:10.457123 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:28:10.745099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:10.745065 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:14.791768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:14.791729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:33.431491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:33.431447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:33.629290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:33.629246 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:40.469988 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:28:40.469942 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:28:40.671540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:40.671495 5043 scope.go:122] "RemoveContainer" containerID="f06c6e509606149b7d8405bb0e6d2c2d3d42752513a275afadd1e67d9bd7b181" Oct 31 23:28:40.683969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:40.683937 5043 scope.go:122] "RemoveContainer" containerID="fef5f76cf68e6867af3954b0c64e9c5e8d39b99ef1daad3a8fd96f4103f84499" Oct 31 23:28:40.694247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:40.694223 5043 scope.go:122] "RemoveContainer" containerID="f1e56a82f231642d94eadd189b600828069e1088432670a668cdbc37a3404225" Oct 31 23:28:40.703174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:40.703151 5043 scope.go:122] "RemoveContainer" containerID="342ece31fb09639319bc0223d0d53d72e365580301a498f53b963b45c9a67227" Oct 31 23:28:40.712128 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:40.712104 5043 scope.go:122] "RemoveContainer" containerID="d48f1afae4c7c4e5d02315dcf4c3103506cade7a97737749f0119cf6c6c9ab1c" Oct 31 23:28:44.684058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:44.684018 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:28:44.823318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:28:44.823287 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:03.908877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:03.908837 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:04.049127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:04.049092 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:10.456303 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:29:10.456262 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:29:14.834491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:14.834278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:15.158146 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:15.158061 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:40.456221 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:29:40.456179 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:29:40.762890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:40.762863 5043 scope.go:122] "RemoveContainer" containerID="c6dffb4bc897d66cc9d638d13d4a567588ca65958fda26121828e98a37966319" Oct 31 23:29:42.466333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:42.466301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:42.698906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:42.698872 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:48.024981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:48.024946 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht"] Oct 31 23:29:48.029915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:48.029885 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn"] Oct 31 23:29:48.032989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:48.032962 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2980884-7kdht"] Oct 31 23:29:48.035311 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:48.035286 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-207ce6f3-zp9dn"] Oct 31 23:29:48.073126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:48.073098 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:29:49.025986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:49.025949 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb"] Oct 31 23:29:49.030511 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:49.030480 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c0a48-5fxsb"] Oct 31 23:29:49.934431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:49.934394 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d723e9b-2145-4fb4-a430-e7c6982b1f5b" path="/var/lib/kubelet/pods/1d723e9b-2145-4fb4-a430-e7c6982b1f5b/volumes" Oct 31 23:29:49.934875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:49.934860 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4272e612-4718-4d0b-99b1-da0907a3865d" path="/var/lib/kubelet/pods/4272e612-4718-4d0b-99b1-da0907a3865d/volumes" Oct 31 23:29:49.935240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:29:49.935228 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74e6297-4386-49eb-bfc6-a86142b3fc2e" path="/var/lib/kubelet/pods/f74e6297-4386-49eb-bfc6-a86142b3fc2e/volumes" Oct 31 23:30:03.036799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:03.036729 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2"] Oct 31 23:30:03.059591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:03.057644 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f1dd1cb-jl4c2"] Oct 31 23:30:03.069567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:03.067167 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9"] Oct 31 23:30:03.077568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:03.075486 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ce14fe-pg7h9"] Oct 31 23:30:03.966569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:03.957905 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30481cd9-96bc-4880-836d-aa92778ce043" path="/var/lib/kubelet/pods/30481cd9-96bc-4880-836d-aa92778ce043/volumes" Oct 31 23:30:03.966569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:03.958518 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4347d5-4a34-4803-9221-7cc587a14089" path="/var/lib/kubelet/pods/7b4347d5-4a34-4803-9221-7cc587a14089/volumes" Oct 31 23:30:06.035962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:06.035926 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt"] Oct 31 23:30:06.051946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:06.051919 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23292ab3-5lhpt"] Oct 31 23:30:07.966874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:07.966842 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bbccd8-dcde-45ab-9522-56a2b73b7218" path="/var/lib/kubelet/pods/12bbccd8-dcde-45ab-9522-56a2b73b7218/volumes" Oct 31 23:30:10.455860 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:30:10.455816 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:30:12.186519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.186487 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg"] Oct 31 23:30:12.186969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.186953 5043 cpu_manager.go:414] "containerMap: removing container" podUID="b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" containerName="revoke-keys" Oct 31 23:30:12.187015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.186970 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" containerName="revoke-keys" Oct 31 23:30:12.187015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.186983 5043 cpu_manager.go:414] "containerMap: removing container" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" Oct 31 23:30:12.187015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.186988 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" Oct 31 23:30:12.187015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187010 5043 cpu_manager.go:414] "containerMap: removing container" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187016 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187036 5043 cpu_manager.go:414] "containerMap: removing container" podUID="74453611-c34e-450c-ad24-bb80abbeaf01" containerName="revoke-keys" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187042 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74453611-c34e-450c-ad24-bb80abbeaf01" containerName="revoke-keys" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187050 5043 cpu_manager.go:414] "containerMap: removing container" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187054 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187121 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="74453611-c34e-450c-ad24-bb80abbeaf01" containerName="revoke-keys" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187130 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1ae77f-51b8-4d9b-875b-daede18c8ab8" containerName="istio-proxy" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187138 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" containerName="revoke-keys" Oct 31 23:30:12.187154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.187146 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" Oct 31 23:30:12.191450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.191432 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.195934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.195874 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:30:12.195934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.195894 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:30:12.196102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.195934 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:30:12.197013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.196995 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:30:12.198244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.198220 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg"] Oct 31 23:30:12.261850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.261815 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n599\" (UniqueName: \"kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.261966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.261859 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.262010 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.261978 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.362843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.362822 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n599\" (UniqueName: \"kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.362960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.362850 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.362960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.362882 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.363503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.363486 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.365188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.365163 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.375308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.375288 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n599\" (UniqueName: \"kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599\") pod \"maas-api-revoke-sub-3d4156f1-9jpxg\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.507294 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.507269 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:12.788082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:12.788057 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg"] Oct 31 23:30:12.790359 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:30:12.790324 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0513ab21_a78d_4bf2_89bb_1651271d9f16.slice/crio-388f6d4da3597ae46d9ccb4d3ce070d57cc350e5a0f5935b4a6635d5d18d4684 WatchSource:0}: Error finding container 388f6d4da3597ae46d9ccb4d3ce070d57cc350e5a0f5935b4a6635d5d18d4684: Status 404 returned error can't find the container with id 388f6d4da3597ae46d9ccb4d3ce070d57cc350e5a0f5935b4a6635d5d18d4684 Oct 31 23:30:13.516592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:13.516557 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" event={"ID":"0513ab21-a78d-4bf2-89bb-1651271d9f16","Type":"ContainerStarted","Data":"f0e3208066fd32c4a86501f375e8bda55d04984c24126d15168d072e5abfdc07"} Oct 31 23:30:13.516993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:13.516647 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" event={"ID":"0513ab21-a78d-4bf2-89bb-1651271d9f16","Type":"ContainerStarted","Data":"388f6d4da3597ae46d9ccb4d3ce070d57cc350e5a0f5935b4a6635d5d18d4684"} Oct 31 23:30:13.535942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:13.535902 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" podStartSLOduration=1.535890169 podStartE2EDuration="1.535890169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:30:13.534783051 +0000 UTC m=+2439.764337464" watchObservedRunningTime="2026-10-31 23:30:13.535890169 +0000 UTC m=+2439.765444579" Oct 31 23:30:14.524101 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:14.524067 5043 generic.go:356] "Generic (PLEG): container finished" podID="0513ab21-a78d-4bf2-89bb-1651271d9f16" containerID="f0e3208066fd32c4a86501f375e8bda55d04984c24126d15168d072e5abfdc07" exitCode=0 Oct 31 23:30:14.524528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:14.524145 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" event={"ID":"0513ab21-a78d-4bf2-89bb-1651271d9f16","Type":"ContainerDied","Data":"f0e3208066fd32c4a86501f375e8bda55d04984c24126d15168d072e5abfdc07"} Oct 31 23:30:14.886681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:14.886607 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:30:15.240286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.240167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:30:15.681462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.681434 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:15.795814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.795730 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599\" (UniqueName: \"kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599\") pod \"0513ab21-a78d-4bf2-89bb-1651271d9f16\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " Oct 31 23:30:15.795994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.795878 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca\") pod \"0513ab21-a78d-4bf2-89bb-1651271d9f16\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " Oct 31 23:30:15.795994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.795924 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca\") pod \"0513ab21-a78d-4bf2-89bb-1651271d9f16\" (UID: \"0513ab21-a78d-4bf2-89bb-1651271d9f16\") " Oct 31 23:30:15.796319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.796296 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca" pod "0513ab21-a78d-4bf2-89bb-1651271d9f16" (UID: "0513ab21-a78d-4bf2-89bb-1651271d9f16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:30:15.797927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.797898 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca" pod "0513ab21-a78d-4bf2-89bb-1651271d9f16" (UID: "0513ab21-a78d-4bf2-89bb-1651271d9f16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:30:15.798026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.797964 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599" pod "0513ab21-a78d-4bf2-89bb-1651271d9f16" (UID: "0513ab21-a78d-4bf2-89bb-1651271d9f16"). InnerVolumeSpecName "kube-api-access-5n599". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:30:15.897058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.897027 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n599\" (UniqueName: \"kubernetes.io/projected/0513ab21-a78d-4bf2-89bb-1651271d9f16-kube-api-access-5n599\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:15.897223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.897060 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:15.897223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:15.897075 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0513ab21-a78d-4bf2-89bb-1651271d9f16-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:16.537994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:16.537964 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" Oct 31 23:30:16.537994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:16.537991 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg" event={"ID":"0513ab21-a78d-4bf2-89bb-1651271d9f16","Type":"ContainerDied","Data":"388f6d4da3597ae46d9ccb4d3ce070d57cc350e5a0f5935b4a6635d5d18d4684"} Oct 31 23:30:16.538205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:16.538033 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="388f6d4da3597ae46d9ccb4d3ce070d57cc350e5a0f5935b4a6635d5d18d4684" Oct 31 23:30:33.382802 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.382764 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl"] Oct 31 23:30:33.383218 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.382997 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" containerID="cri-o://0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c" gracePeriod=30 Oct 31 23:30:33.670336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.670300 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:30:33.671646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.671527 5043 generic.go:356] "Generic (PLEG): container finished" podID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerID="0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c" exitCode=0 Oct 31 23:30:33.671801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.671578 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" event={"ID":"18ae1e98-0054-4490-9a79-56b82a6428a3","Type":"ContainerDied","Data":"0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c"} Oct 31 23:30:33.671801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.671787 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" event={"ID":"18ae1e98-0054-4490-9a79-56b82a6428a3","Type":"ContainerDied","Data":"c12b2ef948b1e207f3213a10f5423891ab1578d917115c9b6db3c3af1fba9fb2"} Oct 31 23:30:33.671909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.671812 5043 scope.go:122] "RemoveContainer" containerID="0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c" Oct 31 23:30:33.683632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.683608 5043 scope.go:122] "RemoveContainer" containerID="0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c" Oct 31 23:30:33.683953 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:30:33.683925 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c\": container with ID starting with 0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c not found: ID does not exist" containerID="0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c" Oct 31 23:30:33.684029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.683953 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c"} err="failed to get container status \"0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c\": rpc error: code = NotFound desc = could not find container \"0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c\": container with ID starting with 0e839d8229ad4cb59205ea241f0b36f8dc26db11527d3e705c71ec7bb875ea8c not found: ID does not exist" Oct 31 23:30:33.777763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.777732 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.777969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.777842 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.777969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.777866 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.777969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.777889 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.777969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.777918 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.777969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.777967 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.778249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.778010 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c\") pod \"18ae1e98-0054-4490-9a79-56b82a6428a3\" (UID: \"18ae1e98-0054-4490-9a79-56b82a6428a3\") " Oct 31 23:30:33.778664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.778623 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:30:33.780903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.780763 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:30:33.780903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.780763 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:30:33.780903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.780793 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:30:33.780903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.780865 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:30:33.781180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.781047 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:30:33.782099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.782070 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c" pod "18ae1e98-0054-4490-9a79-56b82a6428a3" (UID: "18ae1e98-0054-4490-9a79-56b82a6428a3"). InnerVolumeSpecName "kube-api-access-v4k5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:30:33.879367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.879324 5043 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-dns-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:33.879990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.879963 5043 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-csr-ca-configmap\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:33.880153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.880133 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:33.880289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.880265 5043 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ae1e98-0054-4490-9a79-56b82a6428a3-local-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:33.880418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.880403 5043 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-istio-kubeconfig\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:33.880522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.880508 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k5c\" (UniqueName: \"kubernetes.io/projected/18ae1e98-0054-4490-9a79-56b82a6428a3-kube-api-access-v4k5c\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:33.880863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:33.880838 5043 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18ae1e98-0054-4490-9a79-56b82a6428a3-cacerts\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:34.683328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:34.683290 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl" Oct 31 23:30:34.708950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:34.708922 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl"] Oct 31 23:30:34.713670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:34.713640 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xbgvl"] Oct 31 23:30:35.945568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:35.944727 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" path="/var/lib/kubelet/pods/18ae1e98-0054-4490-9a79-56b82a6428a3/volumes" Oct 31 23:30:38.700051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:38.700016 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:30:40.462704 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:30:40.462663 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:30:40.701116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.701084 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:30:40.791508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.791477 5043 scope.go:122] "RemoveContainer" containerID="4209a629a4c3b60feb396533d4e6097b4f625f6d9cca941c5ad90dc43424dca1" Oct 31 23:30:40.800901 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.800882 5043 scope.go:122] "RemoveContainer" containerID="56a2ddc34092de7d21ecdcb1115e783d235e7e479f8c99a4148a5dd96d00194a" Oct 31 23:30:40.809312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.809290 5043 scope.go:122] "RemoveContainer" containerID="61584c5fc7c47a20f74390d78f444e55032a49cbb0b48398c6a8988194d7f52e" Oct 31 23:30:40.817944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.817926 5043 scope.go:122] "RemoveContainer" containerID="537cdf640cd9d329c940d4e516882aecb9a112aca25a1a18d2dbd11de259ccce" Oct 31 23:30:40.826031 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.826009 5043 scope.go:122] "RemoveContainer" containerID="ada1dcc7626b29492ca802e53f90568ada509f47d91a9179c1e9035e38cb87eb" Oct 31 23:30:40.834094 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.834075 5043 scope.go:122] "RemoveContainer" containerID="a61b815473f03bd142ad2b28ae9df7c9423c7928a316f16d5267449a160524a3" Oct 31 23:30:40.842501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.842483 5043 scope.go:122] "RemoveContainer" containerID="5a445c07d73b4b812a2f53a8e7194a751b4a95531942adccee43e62a48479ebc" Oct 31 23:30:40.850274 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:40.850256 5043 scope.go:122] "RemoveContainer" containerID="747eff1b03fa7c094df4b48984459a803e014769da6a6ecccc30a1090fa6228d" Oct 31 23:30:41.023893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.023862 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr"] Oct 31 23:30:41.030976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.030951 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg"] Oct 31 23:30:41.033361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.033341 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1666a970-97jzr"] Oct 31 23:30:41.035742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.035722 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt"] Oct 31 23:30:41.036166 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036152 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0513ab21-a78d-4bf2-89bb-1651271d9f16" containerName="revoke-keys" Oct 31 23:30:41.036208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036168 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0513ab21-a78d-4bf2-89bb-1651271d9f16" containerName="revoke-keys" Oct 31 23:30:41.036208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036186 5043 cpu_manager.go:414] "containerMap: removing container" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" Oct 31 23:30:41.036208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036193 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" Oct 31 23:30:41.036318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036288 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ae1e98-0054-4490-9a79-56b82a6428a3" containerName="discovery" Oct 31 23:30:41.036318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036297 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="87090fa8-2197-43d2-8bcf-6429c94ef477" containerName="istio-proxy" Oct 31 23:30:41.036318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.036309 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0513ab21-a78d-4bf2-89bb-1651271d9f16" containerName="revoke-keys" Oct 31 23:30:41.041496 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.041479 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022de01f-lw5bg"] Oct 31 23:30:41.041621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.041578 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.045884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.045865 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:30:41.045998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.045880 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt"] Oct 31 23:30:41.045998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.045939 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:30:41.045998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.045987 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:30:41.046140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.046047 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:30:41.196531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.196499 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.196745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.196631 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrvmv\" (UniqueName: \"kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.196797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.196739 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.297379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.297302 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.297379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.297355 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvmv\" (UniqueName: \"kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.297586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.297414 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.298184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.298164 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.299880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.299844 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.310100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.310080 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrvmv\" (UniqueName: \"kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv\") pod \"maas-api-revoke-sub-e2fc9c5b-2gxjt\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.356673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.356652 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:41.639410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.639384 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt"] Oct 31 23:30:41.641289 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:30:41.641260 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bd50d9_91f5_49c0_9a70_35670d10b85c.slice/crio-da49e8055b6fcdfb4f611872495dca0c8f59fcc4428255f064051ecb08a7278c WatchSource:0}: Error finding container da49e8055b6fcdfb4f611872495dca0c8f59fcc4428255f064051ecb08a7278c: Status 404 returned error can't find the container with id da49e8055b6fcdfb4f611872495dca0c8f59fcc4428255f064051ecb08a7278c Oct 31 23:30:41.759480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.759447 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" event={"ID":"a7bd50d9-91f5-49c0-9a70-35670d10b85c","Type":"ContainerStarted","Data":"da49e8055b6fcdfb4f611872495dca0c8f59fcc4428255f064051ecb08a7278c"} Oct 31 23:30:41.936184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.936155 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74453611-c34e-450c-ad24-bb80abbeaf01" path="/var/lib/kubelet/pods/74453611-c34e-450c-ad24-bb80abbeaf01/volumes" Oct 31 23:30:41.936583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:41.936569 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41" path="/var/lib/kubelet/pods/b42ff2ef-0147-4c3f-a0f3-0cb0b1ba4d41/volumes" Oct 31 23:30:42.767071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:42.767043 5043 generic.go:356] "Generic (PLEG): container finished" podID="a7bd50d9-91f5-49c0-9a70-35670d10b85c" containerID="fd355034d0fa7e55f8111aa6506b3926d56128a5873c5536a0442d325c6b7a90" exitCode=0 Oct 31 23:30:42.767464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:42.767087 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" event={"ID":"a7bd50d9-91f5-49c0-9a70-35670d10b85c","Type":"ContainerDied","Data":"fd355034d0fa7e55f8111aa6506b3926d56128a5873c5536a0442d325c6b7a90"} Oct 31 23:30:43.919288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:43.919259 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:44.022013 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.021981 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca\") pod \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " Oct 31 23:30:44.022192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.022043 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca\") pod \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " Oct 31 23:30:44.022192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.022115 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv\" (UniqueName: \"kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv\") pod \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\" (UID: \"a7bd50d9-91f5-49c0-9a70-35670d10b85c\") " Oct 31 23:30:44.022360 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.022332 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca" pod "a7bd50d9-91f5-49c0-9a70-35670d10b85c" (UID: "a7bd50d9-91f5-49c0-9a70-35670d10b85c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:30:44.022534 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.022509 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:44.024349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.024295 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv" pod "a7bd50d9-91f5-49c0-9a70-35670d10b85c" (UID: "a7bd50d9-91f5-49c0-9a70-35670d10b85c"). InnerVolumeSpecName "kube-api-access-nrvmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:30:44.024349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.024322 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca" pod "a7bd50d9-91f5-49c0-9a70-35670d10b85c" (UID: "a7bd50d9-91f5-49c0-9a70-35670d10b85c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:30:44.025970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.025944 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:30:44.123282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.123249 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bd50d9-91f5-49c0-9a70-35670d10b85c-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:44.123282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.123277 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrvmv\" (UniqueName: \"kubernetes.io/projected/a7bd50d9-91f5-49c0-9a70-35670d10b85c-kube-api-access-nrvmv\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:44.162983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.162947 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:30:44.780910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.780881 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" Oct 31 23:30:44.781096 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.780922 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt" event={"ID":"a7bd50d9-91f5-49c0-9a70-35670d10b85c","Type":"ContainerDied","Data":"da49e8055b6fcdfb4f611872495dca0c8f59fcc4428255f064051ecb08a7278c"} Oct 31 23:30:44.781096 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:44.780994 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da49e8055b6fcdfb4f611872495dca0c8f59fcc4428255f064051ecb08a7278c" Oct 31 23:30:51.260909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.260867 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq"] Oct 31 23:30:51.261540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.261522 5043 cpu_manager.go:414] "containerMap: removing container" podUID="a7bd50d9-91f5-49c0-9a70-35670d10b85c" containerName="revoke-keys" Oct 31 23:30:51.261619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.261556 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bd50d9-91f5-49c0-9a70-35670d10b85c" containerName="revoke-keys" Oct 31 23:30:51.261681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.261669 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bd50d9-91f5-49c0-9a70-35670d10b85c" containerName="revoke-keys" Oct 31 23:30:51.265336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.265316 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.271275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.270512 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:30:51.271275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.270601 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:30:51.271275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.270663 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:30:51.271275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.270763 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:30:51.274125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.274096 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq"] Oct 31 23:30:51.288126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.288097 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktb2\" (UniqueName: \"kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.288256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.288151 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.288256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.288239 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.389059 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.389020 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktb2\" (UniqueName: \"kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.389242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.389068 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.389242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.389092 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.389841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.389814 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.391407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.391385 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.401929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.401900 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktb2\" (UniqueName: \"kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2\") pod \"maas-api-revoke-sub-5a29efb3-mgmsq\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.581922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.581845 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:51.862766 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:51.862736 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq"] Oct 31 23:30:51.865575 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:30:51.865526 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ade654_a246_46a4_aee6_4b0898608809.slice/crio-6ebdfd8021661b2d1dec1ba1823008d562cc114f1961a2729d7076877482cfbb WatchSource:0}: Error finding container 6ebdfd8021661b2d1dec1ba1823008d562cc114f1961a2729d7076877482cfbb: Status 404 returned error can't find the container with id 6ebdfd8021661b2d1dec1ba1823008d562cc114f1961a2729d7076877482cfbb Oct 31 23:30:52.834461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:52.834413 5043 generic.go:356] "Generic (PLEG): container finished" podID="31ade654-a246-46a4-aee6-4b0898608809" containerID="3596506453e18880187bfa5731709039d9febfa390c61d3fb007641eac97898f" exitCode=0 Oct 31 23:30:52.834978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:52.834492 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" event={"ID":"31ade654-a246-46a4-aee6-4b0898608809","Type":"ContainerDied","Data":"3596506453e18880187bfa5731709039d9febfa390c61d3fb007641eac97898f"} Oct 31 23:30:52.834978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:52.834581 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" event={"ID":"31ade654-a246-46a4-aee6-4b0898608809","Type":"ContainerStarted","Data":"6ebdfd8021661b2d1dec1ba1823008d562cc114f1961a2729d7076877482cfbb"} Oct 31 23:30:54.606448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.606163 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:30:54.725565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.724888 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca\") pod \"31ade654-a246-46a4-aee6-4b0898608809\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " Oct 31 23:30:54.725565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.725016 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2\" (UniqueName: \"kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2\") pod \"31ade654-a246-46a4-aee6-4b0898608809\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " Oct 31 23:30:54.725565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.725077 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca\") pod \"31ade654-a246-46a4-aee6-4b0898608809\" (UID: \"31ade654-a246-46a4-aee6-4b0898608809\") " Oct 31 23:30:54.736564 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.726315 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca" pod "31ade654-a246-46a4-aee6-4b0898608809" (UID: "31ade654-a246-46a4-aee6-4b0898608809"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:30:54.739607 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.739536 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2" pod "31ade654-a246-46a4-aee6-4b0898608809" (UID: "31ade654-a246-46a4-aee6-4b0898608809"). InnerVolumeSpecName "kube-api-access-7ktb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:30:54.763624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.763598 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca" pod "31ade654-a246-46a4-aee6-4b0898608809" (UID: "31ade654-a246-46a4-aee6-4b0898608809"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:30:54.850044 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.850012 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktb2\" (UniqueName: \"kubernetes.io/projected/31ade654-a246-46a4-aee6-4b0898608809-kube-api-access-7ktb2\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:54.850565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.850531 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ade654-a246-46a4-aee6-4b0898608809-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:54.859623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.859601 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ade654-a246-46a4-aee6-4b0898608809-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:30:54.940595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.914386 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" event={"ID":"31ade654-a246-46a4-aee6-4b0898608809","Type":"ContainerDied","Data":"6ebdfd8021661b2d1dec1ba1823008d562cc114f1961a2729d7076877482cfbb"} Oct 31 23:30:54.940595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.914449 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ebdfd8021661b2d1dec1ba1823008d562cc114f1961a2729d7076877482cfbb" Oct 31 23:30:54.940595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:30:54.914522 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq" Oct 31 23:31:10.455728 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:31:10.455689 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:31:14.946748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:14.946708 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:31:15.131138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:15.131100 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:31:17.663676 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:17.663650 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:31:17.668592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:17.668566 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:31:17.678236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:17.678216 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:31:18.981680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:18.981654 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:31:18.986303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:18.986282 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:31:18.995628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:18.995611 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:31:40.456049 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:31:40.456012 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:31:40.906522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.906484 5043 scope.go:122] "RemoveContainer" containerID="579cfc08fc16c3145b4a14a28ec9c3544567455d156615b2fb9c02e239d51f63" Oct 31 23:31:40.915839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.915812 5043 scope.go:122] "RemoveContainer" containerID="ae9f226a4cabbb473059c912470c648ecc325ab6225c86393f07ce7146bde875" Oct 31 23:31:40.924185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.924168 5043 scope.go:122] "RemoveContainer" containerID="875e3b5b0ef3027b41663f0bb0f0d549fb1db638b3002904340e0f452074d13b" Oct 31 23:31:40.932053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.932036 5043 scope.go:122] "RemoveContainer" containerID="c6750ad76a463853e92597d594e132074dce1836e71fa5b263a079f6d7cd0c67" Oct 31 23:31:40.940317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.940302 5043 scope.go:122] "RemoveContainer" containerID="c028591efecf02c157e8a62a35ec58a357888c5fc982831f138315c029df0a1b" Oct 31 23:31:40.947934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.947911 5043 scope.go:122] "RemoveContainer" containerID="6011a905bc22fdc4078eb0585a55febbbf6ced3c8e3741c46db24279084bccb8" Oct 31 23:31:40.955296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.955280 5043 scope.go:122] "RemoveContainer" containerID="7d6e79ea590e48e8ceea61a5fdb36e8fb9158bd133ba9ac87b792a3480d9eb4d" Oct 31 23:31:40.962883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.962867 5043 scope.go:122] "RemoveContainer" containerID="35eb9d5d236cd699edf3e7e6744d487d9eb71d95d968380e0a6c991f0c2673fd" Oct 31 23:31:40.970309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.970290 5043 scope.go:122] "RemoveContainer" containerID="f5e014abf7b6edaba9a0788a545a9bab5504063dab06b598844184617e9822b7" Oct 31 23:31:40.981147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.981124 5043 scope.go:122] "RemoveContainer" containerID="0c32d3b60fc21517265a73f7e08fb2b1e613677fa037cca5852f44e8be53d1da" Oct 31 23:31:40.988963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.988946 5043 scope.go:122] "RemoveContainer" containerID="1a8337a748a5c171f61a37f35413ba1fe5f6449ce1923dcc072d4d56835262f2" Oct 31 23:31:40.996385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:40.996365 5043 scope.go:122] "RemoveContainer" containerID="d9b869063e60fc567125c87ae1369e33b5ab3bdfc94998891c68996edc3a7248" Oct 31 23:31:41.003468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:31:41.003454 5043 scope.go:122] "RemoveContainer" containerID="c93e12280c32555c8868dcdfbbbce7e7f47da49c88c5b18a0f667833cca27eb6" Oct 31 23:32:10.455600 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:32:10.455526 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:32:14.997417 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:32:14.997333 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:32:15.204653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:32:15.204616 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:32:40.455074 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:32:40.455036 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:33:10.454932 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:33:10.454893 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:33:15.043491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:33:15.043456 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:33:15.231034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:33:15.231002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:33:40.455805 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:33:40.455766 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:34:10.458232 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:34:10.458144 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:34:10.884114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:34:10.884064 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:34:15.134865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:34:15.134833 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:34:15.283353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:34:15.283323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:34:38.019433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:34:38.019382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:34:40.455027 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:34:40.454982 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:35:10.455983 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:35:10.455948 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:35:15.151889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:15.151805 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:35:16.023135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:16.023104 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg"] Oct 31 23:35:16.025391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:16.025368 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9jpxg"] Oct 31 23:35:17.934833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:17.934803 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0513ab21-a78d-4bf2-89bb-1651271d9f16" path="/var/lib/kubelet/pods/0513ab21-a78d-4bf2-89bb-1651271d9f16/volumes" Oct 31 23:35:40.455343 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:35:40.455306 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:35:44.023033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:44.023000 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt"] Oct 31 23:35:44.025425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:44.025403 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-2gxjt"] Oct 31 23:35:45.935132 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:45.935100 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7bd50d9-91f5-49c0-9a70-35670d10b85c" path="/var/lib/kubelet/pods/a7bd50d9-91f5-49c0-9a70-35670d10b85c/volumes" Oct 31 23:35:55.023112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:55.023077 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq"] Oct 31 23:35:55.027116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:55.027089 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-mgmsq"] Oct 31 23:35:55.934880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:35:55.934845 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ade654-a246-46a4-aee6-4b0898608809" path="/var/lib/kubelet/pods/31ade654-a246-46a4-aee6-4b0898608809/volumes" Oct 31 23:36:10.012326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.012270 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh"] Oct 31 23:36:10.012801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.012747 5043 cpu_manager.go:414] "containerMap: removing container" podUID="31ade654-a246-46a4-aee6-4b0898608809" containerName="revoke-keys" Oct 31 23:36:10.012801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.012759 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ade654-a246-46a4-aee6-4b0898608809" containerName="revoke-keys" Oct 31 23:36:10.012875 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.012853 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ade654-a246-46a4-aee6-4b0898608809" containerName="revoke-keys" Oct 31 23:36:10.015928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.015911 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.021254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.021228 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:36:10.021393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.021261 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:36:10.021393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.021275 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:36:10.021393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.021325 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:36:10.038448 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.038422 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh"] Oct 31 23:36:10.160540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.160501 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29cpd\" (UniqueName: \"kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.160750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.160635 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.160750 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.160687 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.262023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.261995 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.262144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.262032 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.262144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.262088 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29cpd\" (UniqueName: \"kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.262807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.262738 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.264455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.264436 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.274859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.274838 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cpd\" (UniqueName: \"kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd\") pod \"maas-api-revoke-sub-3d4156f1-9glhh\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.329148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.329126 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:10.456375 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:36:10.456343 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:36:10.604716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.604676 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh"] Oct 31 23:36:10.607581 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:36:10.607536 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15324d94_ea51_47aa_9576_63e0641818b5.slice/crio-a3d2de56748db4c8cf4146ca6012b504f4cbc4e338471bc7b411f20ec3d98784 WatchSource:0}: Error finding container a3d2de56748db4c8cf4146ca6012b504f4cbc4e338471bc7b411f20ec3d98784: Status 404 returned error can't find the container with id a3d2de56748db4c8cf4146ca6012b504f4cbc4e338471bc7b411f20ec3d98784 Oct 31 23:36:10.653658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:10.653630 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" event={"ID":"15324d94-ea51-47aa-9576-63e0641818b5","Type":"ContainerStarted","Data":"a3d2de56748db4c8cf4146ca6012b504f4cbc4e338471bc7b411f20ec3d98784"} Oct 31 23:36:11.660241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:11.660202 5043 generic.go:356] "Generic (PLEG): container finished" podID="15324d94-ea51-47aa-9576-63e0641818b5" containerID="3d844cc70fd6ec97e6f0759a01aa3d6167bef685546e2342f855298eb0f2651f" exitCode=0 Oct 31 23:36:11.660691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:11.660281 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" event={"ID":"15324d94-ea51-47aa-9576-63e0641818b5","Type":"ContainerDied","Data":"3d844cc70fd6ec97e6f0759a01aa3d6167bef685546e2342f855298eb0f2651f"} Oct 31 23:36:12.801909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.801884 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:12.885463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.885432 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca\") pod \"15324d94-ea51-47aa-9576-63e0641818b5\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " Oct 31 23:36:12.885648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.885498 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca\") pod \"15324d94-ea51-47aa-9576-63e0641818b5\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " Oct 31 23:36:12.885648 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.885591 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd\" (UniqueName: \"kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd\") pod \"15324d94-ea51-47aa-9576-63e0641818b5\" (UID: \"15324d94-ea51-47aa-9576-63e0641818b5\") " Oct 31 23:36:12.885910 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.885881 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca" pod "15324d94-ea51-47aa-9576-63e0641818b5" (UID: "15324d94-ea51-47aa-9576-63e0641818b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:12.887716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.887690 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd" pod "15324d94-ea51-47aa-9576-63e0641818b5" (UID: "15324d94-ea51-47aa-9576-63e0641818b5"). InnerVolumeSpecName "kube-api-access-29cpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:12.887716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.887695 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca" pod "15324d94-ea51-47aa-9576-63e0641818b5" (UID: "15324d94-ea51-47aa-9576-63e0641818b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:36:12.987078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.987009 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29cpd\" (UniqueName: \"kubernetes.io/projected/15324d94-ea51-47aa-9576-63e0641818b5-kube-api-access-29cpd\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:12.987078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.987036 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:12.987078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:12.987047 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15324d94-ea51-47aa-9576-63e0641818b5-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:13.672588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:13.672558 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" Oct 31 23:36:13.672588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:13.672574 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh" event={"ID":"15324d94-ea51-47aa-9576-63e0641818b5","Type":"ContainerDied","Data":"a3d2de56748db4c8cf4146ca6012b504f4cbc4e338471bc7b411f20ec3d98784"} Oct 31 23:36:13.672856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:13.672621 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3d2de56748db4c8cf4146ca6012b504f4cbc4e338471bc7b411f20ec3d98784" Oct 31 23:36:15.236214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:15.236184 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:17.704384 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:17.704357 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:36:17.709214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:17.709191 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:36:17.718349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:17.718331 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:36:19.028222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:19.028195 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:36:19.032568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:19.032533 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:36:19.041340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:19.041323 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:36:19.780731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:19.780696 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs"] Oct 31 23:36:19.780995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:19.780939 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" containerID="cri-o://240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436" gracePeriod=30 Oct 31 23:36:25.041351 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.041316 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 23:36:25.127451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127362 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127417 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127481 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127528 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127583 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127638 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127688 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127722 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.127976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.127802 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket\") pod \"11dc44da-2acd-4370-a0c7-37b534779995\" (UID: \"11dc44da-2acd-4370-a0c7-37b534779995\") " Oct 31 23:36:25.128431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.128402 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:25.128841 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.128814 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:25.128963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.128843 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:25.129372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.129324 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:25.129372 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.129324 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:25.131614 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.131584 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:25.132143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.132120 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:36:25.132244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.132124 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:25.132244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.132181 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl" pod "11dc44da-2acd-4370-a0c7-37b534779995" (UID: "11dc44da-2acd-4370-a0c7-37b534779995"). InnerVolumeSpecName "kube-api-access-4mzdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:25.229599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229537 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229594 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mzdl\" (UniqueName: \"kubernetes.io/projected/11dc44da-2acd-4370-a0c7-37b534779995-kube-api-access-4mzdl\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229606 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dc44da-2acd-4370-a0c7-37b534779995-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229616 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229625 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229634 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dc44da-2acd-4370-a0c7-37b534779995-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229642 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229649 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.229839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.229659 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dc44da-2acd-4370-a0c7-37b534779995-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:25.787772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.787737 5043 generic.go:356] "Generic (PLEG): container finished" podID="11dc44da-2acd-4370-a0c7-37b534779995" containerID="240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436" exitCode=0 Oct 31 23:36:25.787965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.787789 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" Oct 31 23:36:25.787965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.787823 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" event={"ID":"11dc44da-2acd-4370-a0c7-37b534779995","Type":"ContainerDied","Data":"240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436"} Oct 31 23:36:25.787965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.787888 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs" event={"ID":"11dc44da-2acd-4370-a0c7-37b534779995","Type":"ContainerDied","Data":"a50680158a4600e5f5ddc4ba9f70be60a759c7b6964b5970e9e2cba7c9b3ec0c"} Oct 31 23:36:25.787965 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.787920 5043 scope.go:122] "RemoveContainer" containerID="240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436" Oct 31 23:36:25.799722 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.799704 5043 scope.go:122] "RemoveContainer" containerID="c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3" Oct 31 23:36:25.813709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.813683 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs"] Oct 31 23:36:25.815037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.815017 5043 scope.go:122] "RemoveContainer" containerID="240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436" Oct 31 23:36:25.815332 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:36:25.815311 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436\": container with ID starting with 240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436 not found: ID does not exist" containerID="240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436" Oct 31 23:36:25.815404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.815341 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436"} err="failed to get container status \"240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436\": rpc error: code = NotFound desc = could not find container \"240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436\": container with ID starting with 240a9f074490482934aa90e954d1570284b86283fc8fe1c6bfc8baaa0d81f436 not found: ID does not exist" Oct 31 23:36:25.815404 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.815359 5043 scope.go:122] "RemoveContainer" containerID="c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3" Oct 31 23:36:25.815627 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:36:25.815606 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3\": container with ID starting with c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3 not found: ID does not exist" containerID="c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3" Oct 31 23:36:25.815699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.815636 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3"} err="failed to get container status \"c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3\": rpc error: code = NotFound desc = could not find container \"c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3\": container with ID starting with c2d4761a9404eb63250d0a71d789b3ae068bd362d94eabba84e637075b45cfc3 not found: ID does not exist" Oct 31 23:36:25.818932 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.818911 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-x9wxs"] Oct 31 23:36:25.935740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:25.935706 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dc44da-2acd-4370-a0c7-37b534779995" path="/var/lib/kubelet/pods/11dc44da-2acd-4370-a0c7-37b534779995/volumes" Oct 31 23:36:28.389779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.389747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:28.792178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792136 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq"] Oct 31 23:36:28.792698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792672 5043 cpu_manager.go:414] "containerMap: removing container" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" Oct 31 23:36:28.792698 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792692 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" Oct 31 23:36:28.792882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792724 5043 cpu_manager.go:414] "containerMap: removing container" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" Oct 31 23:36:28.792882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792731 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" Oct 31 23:36:28.792882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792776 5043 cpu_manager.go:414] "containerMap: removing container" podUID="15324d94-ea51-47aa-9576-63e0641818b5" containerName="revoke-keys" Oct 31 23:36:28.792882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792784 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15324d94-ea51-47aa-9576-63e0641818b5" containerName="revoke-keys" Oct 31 23:36:28.793088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792883 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" Oct 31 23:36:28.793088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.792902 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="15324d94-ea51-47aa-9576-63e0641818b5" containerName="revoke-keys" Oct 31 23:36:28.797760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.797730 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.805977 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.805955 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c4619f-openshift-default-dockercfg-zq7hf\"" Oct 31 23:36:28.810100 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.810079 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq"] Oct 31 23:36:28.865429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865399 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865434 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865459 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865627 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865655 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865777 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865727 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865791 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbgw\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865824 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.865916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.865910 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.882040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.881963 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:28.966855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.966819 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbgw\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.966855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.966859 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.966912 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.966957 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.966983 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967007 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967043 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967072 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967115 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967099 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967442 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967495 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967689 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967721 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.967949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.967790 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.969453 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.969426 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.969638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.969613 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.979826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.979805 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:28.980034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:28.980012 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbgw\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw\") pod \"e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:29.023378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.023339 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:29.117516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.117423 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:29.405907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.405880 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq"] Oct 31 23:36:29.408015 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:36:29.407974 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3da7da_e58f_41cc_99ea_03e12f949416.slice/crio-eb81bc13f130d97eaea0830b65cada2005654d7619d081d2fcfff19f04d72e14 WatchSource:0}: Error finding container eb81bc13f130d97eaea0830b65cada2005654d7619d081d2fcfff19f04d72e14: Status 404 returned error can't find the container with id eb81bc13f130d97eaea0830b65cada2005654d7619d081d2fcfff19f04d72e14 Oct 31 23:36:29.410490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.410458 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:36:29.410589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.410521 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:36:29.410589 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.410583 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:36:29.814087 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.814046 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" event={"ID":"7c3da7da-e58f-41cc-99ea-03e12f949416","Type":"ContainerStarted","Data":"81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab"} Oct 31 23:36:29.814243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.814108 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" event={"ID":"7c3da7da-e58f-41cc-99ea-03e12f949416","Type":"ContainerStarted","Data":"eb81bc13f130d97eaea0830b65cada2005654d7619d081d2fcfff19f04d72e14"} Oct 31 23:36:29.840973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:29.840923 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podStartSLOduration=1.8409084789999999 podStartE2EDuration="1.840908479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:29.838327766 +0000 UTC m=+2816.067882188" watchObservedRunningTime="2026-10-31 23:36:29.840908479 +0000 UTC m=+2816.070462889" Oct 31 23:36:30.117994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:30.117903 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:30.123667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:30.123639 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:30.820829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:30.820789 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:30.821891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:30.821866 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:32.148131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:32.148099 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:32.289934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:32.289906 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:33.219849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.219817 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w"] Oct 31 23:36:33.220507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.220484 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dc44da-2acd-4370-a0c7-37b534779995" containerName="istio-proxy" Oct 31 23:36:33.223808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.223787 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.235304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.235224 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w"] Oct 31 23:36:33.235937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.235909 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:36:33.236098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.236001 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:36:33.254056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.254029 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c4619f-serving-cert\"" Oct 31 23:36:33.254253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.254160 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 23:36:33.254253 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.254221 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7j72\"" Oct 31 23:36:33.312321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.312283 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96h6d\" (UniqueName: \"kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.312508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.312396 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.312508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.312450 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.312508 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.312470 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.316180 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.316151 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:33.413817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.413785 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.413978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.413835 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.413978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.413862 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.413978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.413932 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96h6d\" (UniqueName: \"kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.414480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.414451 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.416574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.416524 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.416708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.416574 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.426321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.426291 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96h6d\" (UniqueName: \"kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d\") pod \"maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.461773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.461526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:33.541762 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.541733 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:33.822866 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.822835 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w"] Oct 31 23:36:33.824667 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:36:33.824636 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0a659f_44a9_4e4d_aa99_dec4dc51c81c.slice/crio-64b5fa372cd2066dce7c5d2d6cf18dc4560965cb6315f11e07e63969ea3aad0f WatchSource:0}: Error finding container 64b5fa372cd2066dce7c5d2d6cf18dc4560965cb6315f11e07e63969ea3aad0f: Status 404 returned error can't find the container with id 64b5fa372cd2066dce7c5d2d6cf18dc4560965cb6315f11e07e63969ea3aad0f Oct 31 23:36:33.825997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.825979 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:36:33.837229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:33.837203 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" event={"ID":"df0a659f-44a9-4e4d-aa99-dec4dc51c81c","Type":"ContainerStarted","Data":"64b5fa372cd2066dce7c5d2d6cf18dc4560965cb6315f11e07e63969ea3aad0f"} Oct 31 23:36:34.231716 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.231610 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:34.798047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.797994 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb"] Oct 31 23:36:34.798588 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.798227 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" containerID="cri-o://74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b" gracePeriod=30 Oct 31 23:36:34.800502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.800479 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl"] Oct 31 23:36:34.800693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.800673 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" containerID="cri-o://15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56" gracePeriod=30 Oct 31 23:36:34.844472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.844431 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" event={"ID":"df0a659f-44a9-4e4d-aa99-dec4dc51c81c","Type":"ContainerStarted","Data":"87cdcadbf56376312bb5e580c6157b6561245c0583d9018dae0ee9174dc14128"} Oct 31 23:36:34.844680 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.844514 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:34.883537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:34.883470 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" podStartSLOduration=1.3447707850000001 podStartE2EDuration="1.883453014s" totalImagesPullingTime="538.682229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:36:34.880909079 +0000 UTC m=+2821.110463531" watchObservedRunningTime="2026-10-31 23:36:34.883453014 +0000 UTC m=+2821.113007424" Oct 31 23:36:35.020232 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:35.020182 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:35.618406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:35.618259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:36.336155 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:36.336123 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:37.020566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:37.019050 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:37.622576 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:37.620999 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:38.221569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:38.219145 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:38.825164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:38.825127 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:39.411226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:39.410983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:40.061301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.061173 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:40.160838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.160499 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:36:40.200167 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.200124 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:36:40.201943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.201624 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.201943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.201906 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202161 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.202507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202470 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.202507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202484 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:40.202752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202586 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.202752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202627 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.202752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202659 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.202911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202771 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.202911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202823 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.202911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202874 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.203055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.202910 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs\") pod \"1d96328a-6b55-4673-9ad7-480e8b36dc39\" (UID: \"1d96328a-6b55-4673-9ad7-480e8b36dc39\") " Oct 31 23:36:40.203667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.203251 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.203667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.203275 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.203667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.203289 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d96328a-6b55-4673-9ad7-480e8b36dc39-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.203667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.203364 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.203966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.203774 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.206075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.205926 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.206075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.206043 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "kube-api-access-ftvr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.206343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.206320 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.207416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.207393 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo" pod "1d96328a-6b55-4673-9ad7-480e8b36dc39" (UID: "1d96328a-6b55-4673-9ad7-480e8b36dc39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:36:40.304026 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.303992 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304041 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304062 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304138 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304203 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304195 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304236 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304291 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304320 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304338 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.304462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304355 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket\") pod \"796f8080-e834-4e54-aaca-808246fe73e5\" (UID: \"796f8080-e834-4e54-aaca-808246fe73e5\") " Oct 31 23:36:40.304462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304450 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.304799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304625 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.304799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304699 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:36:40.304799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304724 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.304909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304833 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.304909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304860 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.304909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304874 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304916 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304930 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d96328a-6b55-4673-9ad7-480e8b36dc39-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304944 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796f8080-e834-4e54-aaca-808246fe73e5-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304957 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304970 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d96328a-6b55-4673-9ad7-480e8b36dc39-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.304984 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftvr8\" (UniqueName: \"kubernetes.io/projected/1d96328a-6b55-4673-9ad7-480e8b36dc39-kube-api-access-ftvr8\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.305000 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.305062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.305013 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.306658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.306632 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.306658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.306648 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "kube-api-access-nbrkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:36:40.306828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.306743 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:36:40.306971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.306948 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy" pod "796f8080-e834-4e54-aaca-808246fe73e5" (UID: "796f8080-e834-4e54-aaca-808246fe73e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:36:40.405710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.405631 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.405710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.405660 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796f8080-e834-4e54-aaca-808246fe73e5-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.405710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.405673 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796f8080-e834-4e54-aaca-808246fe73e5-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.405710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.405685 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrkj\" (UniqueName: \"kubernetes.io/projected/796f8080-e834-4e54-aaca-808246fe73e5-kube-api-access-nbrkj\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:36:40.472523 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:36:40.472486 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:36:40.761903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.761868 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:40.855298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.855257 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:36:40.883667 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.883635 5043 generic.go:356] "Generic (PLEG): container finished" podID="796f8080-e834-4e54-aaca-808246fe73e5" containerID="15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56" exitCode=0 Oct 31 23:36:40.883834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.883683 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" Oct 31 23:36:40.883834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.883716 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" event={"ID":"796f8080-e834-4e54-aaca-808246fe73e5","Type":"ContainerDied","Data":"15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56"} Oct 31 23:36:40.883834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.883767 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl" event={"ID":"796f8080-e834-4e54-aaca-808246fe73e5","Type":"ContainerDied","Data":"abfa7fb36ee336ec31fc3288dd93e6e980c60103dc3530aeffeef432a202608a"} Oct 31 23:36:40.883834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.883796 5043 scope.go:122] "RemoveContainer" containerID="15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56" Oct 31 23:36:40.886590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.885637 5043 generic.go:356] "Generic (PLEG): container finished" podID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerID="74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b" exitCode=0 Oct 31 23:36:40.886590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.885671 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" event={"ID":"1d96328a-6b55-4673-9ad7-480e8b36dc39","Type":"ContainerDied","Data":"74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b"} Oct 31 23:36:40.886590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.885710 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" event={"ID":"1d96328a-6b55-4673-9ad7-480e8b36dc39","Type":"ContainerDied","Data":"b649ccf1b6364a92e490f51682abee097ec988b07e7bea1c56c9dd629db641e9"} Oct 31 23:36:40.886590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.885786 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb" Oct 31 23:36:40.904483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.904446 5043 scope.go:122] "RemoveContainer" containerID="15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56" Oct 31 23:36:40.905178 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:36:40.905154 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56\": container with ID starting with 15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56 not found: ID does not exist" containerID="15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56" Oct 31 23:36:40.905279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.905182 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56"} err="failed to get container status \"15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56\": rpc error: code = NotFound desc = could not find container \"15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56\": container with ID starting with 15ad4cb18e104192f134f5a6e730d5c453046a0e812668a1c5b731d6c143dd56 not found: ID does not exist" Oct 31 23:36:40.905279 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.905199 5043 scope.go:122] "RemoveContainer" containerID="74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b" Oct 31 23:36:40.942603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.941774 5043 scope.go:122] "RemoveContainer" containerID="74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b" Oct 31 23:36:40.942603 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:36:40.942209 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b\": container with ID starting with 74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b not found: ID does not exist" containerID="74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b" Oct 31 23:36:40.942603 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.942246 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b"} err="failed to get container status \"74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b\": rpc error: code = NotFound desc = could not find container \"74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b\": container with ID starting with 74cd6e2b02b37ff6a828c649b2f5c3d057758b26df2105b6bc23ac7b7af2db0b not found: ID does not exist" Oct 31 23:36:40.945491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.945459 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl"] Oct 31 23:36:40.947911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.947888 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-p9wjl"] Oct 31 23:36:40.961902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.961846 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb"] Oct 31 23:36:40.966356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:40.966331 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jqhtb"] Oct 31 23:36:41.099768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:41.099696 5043 scope.go:122] "RemoveContainer" containerID="f0e3208066fd32c4a86501f375e8bda55d04984c24126d15168d072e5abfdc07" Oct 31 23:36:41.411615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:41.411502 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:41.953022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:41.952985 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" path="/var/lib/kubelet/pods/1d96328a-6b55-4673-9ad7-480e8b36dc39/volumes" Oct 31 23:36:41.953439 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:41.953424 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796f8080-e834-4e54-aaca-808246fe73e5" path="/var/lib/kubelet/pods/796f8080-e834-4e54-aaca-808246fe73e5/volumes" Oct 31 23:36:42.070502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:42.070471 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:42.721199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:42.721113 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:43.303518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:43.303484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:44.057652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:44.057616 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:44.613095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:44.612298 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:45.328004 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.327979 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:45.822420 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.822379 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 23:36:45.822826 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.822432 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 23:36:45.915114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.915081 5043 generic.go:356] "Generic (PLEG): container finished" podID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerID="81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab" exitCode=0 Oct 31 23:36:45.915304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.915155 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" event={"ID":"7c3da7da-e58f-41cc-99ea-03e12f949416","Type":"ContainerDied","Data":"81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab"} Oct 31 23:36:45.915579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.915541 5043 scope.go:122] "RemoveContainer" containerID="81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab" Oct 31 23:36:45.916731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.916699 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:36:45.916808 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.916771 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:36:45.916847 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:45.916810 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:36:46.015358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:46.015327 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:46.632925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:46.632873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:46.943740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:46.943648 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" event={"ID":"7c3da7da-e58f-41cc-99ea-03e12f949416","Type":"ContainerStarted","Data":"90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394"} Oct 31 23:36:47.132569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.130001 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:47.132569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.130133 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 23:36:47.132569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.130178 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 23:36:47.235997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.235898 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:47.434581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.433016 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:47.839957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.839919 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:47.975291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:47.975262 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:48.108829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:48.108752 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:48.119347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:48.118933 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 23:36:48.119347 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:48.118988 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 23:36:48.329582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:48.328645 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:49.045812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:49.045724 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:49.136568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:49.117916 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 23:36:49.136568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:49.117985 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 23:36:49.182406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:49.182370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:49.611886 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:49.611852 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:50.119571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:50.118667 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 23:36:50.119571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:50.118766 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 23:36:50.291611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:50.291574 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:51.013869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:51.013835 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:51.122569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:51.119946 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 23:36:51.122569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:51.120024 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 23:36:51.854066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:51.854031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:52.143863 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:52.143779 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:52.144280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:52.144169 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:52.144978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:52.144954 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:36:52.516429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:52.516394 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:53.330328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:53.330288 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:54.038912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:54.038877 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:54.826977 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:54.826936 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:55.522189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:55.522158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:56.236851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:56.236817 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:56.912538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:56.912504 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:57.502414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:57.502367 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:58.207830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:58.207793 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:58.893476 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:58.893438 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:36:59.503296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:36:59.503259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:00.266501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:00.266468 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:00.902397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:00.902364 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:01.502357 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:01.502323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:02.208490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:02.208446 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:02.361987 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:02.361950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:02.502316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:02.502284 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:02.797318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:02.797243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:03.407672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:03.407641 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:04.005624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:04.005582 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:04.619215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:04.619181 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:05.320825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:05.320792 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:05.970292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:05.970254 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:06.606426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:06.606395 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:07.235318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:07.235276 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:07.901494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:07.901428 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:08.509418 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:08.509356 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:09.102838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:09.102801 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:09.709162 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:09.709128 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:10.304970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:10.304937 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:10.456105 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:37:10.456065 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:37:10.914559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:10.914513 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:11.623224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:11.623194 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:12.202788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:12.202751 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:12.902958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:12.902926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:13.505185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:13.505125 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:14.112609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:14.112573 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:14.713185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:14.713141 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:15.247616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:15.247581 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:15.384145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:15.384111 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:15.916861 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:15.916809 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:16.509027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:16.508993 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:17.126260 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:17.126225 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:17.714121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:17.714089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:18.334850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:18.334818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:18.960520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:18.960484 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:19.708723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:19.708689 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:20.320506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:20.320412 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:20.909361 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:20.909328 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:21.583879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:21.583837 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:22.297783 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:22.297740 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:22.899855 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:22.899800 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:23.506768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:23.506712 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:24.111290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:24.111245 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:24.707985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:24.707948 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:25.309857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:25.309815 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:25.935998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:25.935968 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:26.174982 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:26.174950 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:26.503789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:26.503747 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:27.103170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:27.103130 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:27.723277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:27.723243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:28.412567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:28.412513 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:29.020423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:29.020388 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:29.867447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:29.867379 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:30.138869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:30.138792 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:30.498289 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:30.498256 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:31.199572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:31.199527 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:31.864896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:31.864856 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:32.618642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:32.618579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:33.285724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:33.285682 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:33.914885 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:33.914850 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:34.509880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:34.509838 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:35.154352 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:35.154315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:35.288592 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:35.288556 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:35.708104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:35.708059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:36.298909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:36.298877 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:36.905962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:36.905926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:37.545792 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:37.545759 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:38.108644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:38.108540 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:38.737251 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:38.737219 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:39.313672 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:39.313601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:39.913017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:39.912983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:40.457238 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:37:40.457198 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:37:40.529506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:40.529467 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:40.665225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:40.665187 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:41.020478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.020447 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w"] Oct 31 23:37:41.020984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.020716 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" podUID="df0a659f-44a9-4e4d-aa99-dec4dc51c81c" containerName="maas-api" containerID="cri-o://87cdcadbf56376312bb5e580c6157b6561245c0583d9018dae0ee9174dc14128" gracePeriod=30 Oct 31 23:37:41.113595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.113562 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:41.164455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.164420 5043 scope.go:122] "RemoveContainer" containerID="3596506453e18880187bfa5731709039d9febfa390c61d3fb007641eac97898f" Oct 31 23:37:41.181161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.181140 5043 scope.go:122] "RemoveContainer" containerID="fd355034d0fa7e55f8111aa6506b3926d56128a5873c5536a0442d325c6b7a90" Oct 31 23:37:41.228772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.228733 5043 generic.go:356] "Generic (PLEG): container finished" podID="df0a659f-44a9-4e4d-aa99-dec4dc51c81c" containerID="87cdcadbf56376312bb5e580c6157b6561245c0583d9018dae0ee9174dc14128" exitCode=0 Oct 31 23:37:41.228976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.228954 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" event={"ID":"df0a659f-44a9-4e4d-aa99-dec4dc51c81c","Type":"ContainerDied","Data":"87cdcadbf56376312bb5e580c6157b6561245c0583d9018dae0ee9174dc14128"} Oct 31 23:37:41.249839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.249810 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:41.284813 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.284672 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:37:41.385632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.385586 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:41.428974 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.428943 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca\") pod \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " Oct 31 23:37:41.429148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.429044 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls\") pod \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " Oct 31 23:37:41.429148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.429075 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d\" (UniqueName: \"kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d\") pod \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " Oct 31 23:37:41.429148 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.429125 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs\") pod \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\" (UID: \"df0a659f-44a9-4e4d-aa99-dec4dc51c81c\") " Oct 31 23:37:41.429395 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.429369 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca" pod "df0a659f-44a9-4e4d-aa99-dec4dc51c81c" (UID: "df0a659f-44a9-4e4d-aa99-dec4dc51c81c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:37:41.431273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.431237 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls" pod "df0a659f-44a9-4e4d-aa99-dec4dc51c81c" (UID: "df0a659f-44a9-4e4d-aa99-dec4dc51c81c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:37:41.431699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.431677 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs" pod "df0a659f-44a9-4e4d-aa99-dec4dc51c81c" (UID: "df0a659f-44a9-4e4d-aa99-dec4dc51c81c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:37:41.431763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.431743 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d" pod "df0a659f-44a9-4e4d-aa99-dec4dc51c81c" (UID: "df0a659f-44a9-4e4d-aa99-dec4dc51c81c"). InnerVolumeSpecName "kube-api-access-96h6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:37:41.530451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.530420 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:41.530451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.530452 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:41.530679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.530466 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:41.530679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.530479 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96h6d\" (UniqueName: \"kubernetes.io/projected/df0a659f-44a9-4e4d-aa99-dec4dc51c81c-kube-api-access-96h6d\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:41.805969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:41.805932 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:42.236223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:42.236146 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" Oct 31 23:37:42.236714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:42.236146 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w" event={"ID":"df0a659f-44a9-4e4d-aa99-dec4dc51c81c","Type":"ContainerDied","Data":"64b5fa372cd2066dce7c5d2d6cf18dc4560965cb6315f11e07e63969ea3aad0f"} Oct 31 23:37:42.236714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:42.236307 5043 scope.go:122] "RemoveContainer" containerID="87cdcadbf56376312bb5e580c6157b6561245c0583d9018dae0ee9174dc14128" Oct 31 23:37:42.255525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:42.255502 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w"] Oct 31 23:37:42.261124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:42.261104 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c4619f-58fb989d7f-5nq8w"] Oct 31 23:37:43.935082 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:43.935050 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0a659f-44a9-4e4d-aa99-dec4dc51c81c" path="/var/lib/kubelet/pods/df0a659f-44a9-4e4d-aa99-dec4dc51c81c/volumes" Oct 31 23:37:47.512281 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:47.512243 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:47.712931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:47.712898 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:47.793220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:47.793139 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq"] Oct 31 23:37:47.793413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:47.793387 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" containerID="cri-o://90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394" gracePeriod=30 Oct 31 23:37:47.875678 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:47.875646 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:37:49.112902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.112869 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh"] Oct 31 23:37:49.118449 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.118424 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9glhh"] Oct 31 23:37:49.135391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135364 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c"] Oct 31 23:37:49.135852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135837 5043 cpu_manager.go:414] "containerMap: removing container" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" Oct 31 23:37:49.135907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135854 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" Oct 31 23:37:49.135907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135868 5043 cpu_manager.go:414] "containerMap: removing container" podUID="df0a659f-44a9-4e4d-aa99-dec4dc51c81c" containerName="maas-api" Oct 31 23:37:49.135907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135873 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0a659f-44a9-4e4d-aa99-dec4dc51c81c" containerName="maas-api" Oct 31 23:37:49.135907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135883 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" Oct 31 23:37:49.135907 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135888 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" Oct 31 23:37:49.136069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135982 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="796f8080-e834-4e54-aaca-808246fe73e5" containerName="istio-proxy" Oct 31 23:37:49.136069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.135993 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0a659f-44a9-4e4d-aa99-dec4dc51c81c" containerName="maas-api" Oct 31 23:37:49.136069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.136000 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d96328a-6b55-4673-9ad7-480e8b36dc39" containerName="istio-proxy" Oct 31 23:37:49.140406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.140389 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.146017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.145996 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:37:49.146245 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.146228 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:37:49.148109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.148083 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c"] Oct 31 23:37:49.188297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.188271 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:37:49.188424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.188350 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:37:49.298412 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.298379 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6rp\" (UniqueName: \"kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.298591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.298522 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.298591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.298583 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.399891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.399816 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.399891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.399857 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.400075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.399899 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6rp\" (UniqueName: \"kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.400637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.400614 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.402229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.402204 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.412158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.412135 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6rp\" (UniqueName: \"kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp\") pod \"maas-api-revoke-sub-3d4156f1-z9g6c\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.459035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.459012 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:49.735005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.734964 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c"] Oct 31 23:37:49.737437 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:37:49.737412 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286dd146_aa4a_4d43_9324_eae87b6bfc88.slice/crio-7935557fb8ede1a68cc7ea14b73e2890939d88c129923eefc6e5fe6009621350 WatchSource:0}: Error finding container 7935557fb8ede1a68cc7ea14b73e2890939d88c129923eefc6e5fe6009621350: Status 404 returned error can't find the container with id 7935557fb8ede1a68cc7ea14b73e2890939d88c129923eefc6e5fe6009621350 Oct 31 23:37:49.934738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:49.934710 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15324d94-ea51-47aa-9576-63e0641818b5" path="/var/lib/kubelet/pods/15324d94-ea51-47aa-9576-63e0641818b5/volumes" Oct 31 23:37:50.284267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:50.284184 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" event={"ID":"286dd146-aa4a-4d43-9324-eae87b6bfc88","Type":"ContainerStarted","Data":"e7e4c4f88dc4d865c4d2b3b7afa038637a7a19de0f63100c4e6d2afb9c8cb6ae"} Oct 31 23:37:50.284267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:50.284249 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" event={"ID":"286dd146-aa4a-4d43-9324-eae87b6bfc88","Type":"ContainerStarted","Data":"7935557fb8ede1a68cc7ea14b73e2890939d88c129923eefc6e5fe6009621350"} Oct 31 23:37:50.303540 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:50.303497 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" podStartSLOduration=1.303483998 podStartE2EDuration="1.303483998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:37:50.302093985 +0000 UTC m=+2896.531648395" watchObservedRunningTime="2026-10-31 23:37:50.303483998 +0000 UTC m=+2896.533038445" Oct 31 23:37:51.291745 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:51.291713 5043 generic.go:356] "Generic (PLEG): container finished" podID="286dd146-aa4a-4d43-9324-eae87b6bfc88" containerID="e7e4c4f88dc4d865c4d2b3b7afa038637a7a19de0f63100c4e6d2afb9c8cb6ae" exitCode=0 Oct 31 23:37:51.292150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:51.291808 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" event={"ID":"286dd146-aa4a-4d43-9324-eae87b6bfc88","Type":"ContainerDied","Data":"e7e4c4f88dc4d865c4d2b3b7afa038637a7a19de0f63100c4e6d2afb9c8cb6ae"} Oct 31 23:37:52.146500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.146462 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:37:52.146711 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.146528 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:37:52.433908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.433885 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:52.529983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.529948 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp\" (UniqueName: \"kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp\") pod \"286dd146-aa4a-4d43-9324-eae87b6bfc88\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " Oct 31 23:37:52.530173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.530041 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca\") pod \"286dd146-aa4a-4d43-9324-eae87b6bfc88\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " Oct 31 23:37:52.530173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.530096 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca\") pod \"286dd146-aa4a-4d43-9324-eae87b6bfc88\" (UID: \"286dd146-aa4a-4d43-9324-eae87b6bfc88\") " Oct 31 23:37:52.530383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.530362 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca" pod "286dd146-aa4a-4d43-9324-eae87b6bfc88" (UID: "286dd146-aa4a-4d43-9324-eae87b6bfc88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:37:52.532051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.532027 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp" pod "286dd146-aa4a-4d43-9324-eae87b6bfc88" (UID: "286dd146-aa4a-4d43-9324-eae87b6bfc88"). InnerVolumeSpecName "kube-api-access-zr6rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:37:52.532205 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.532187 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca" pod "286dd146-aa4a-4d43-9324-eae87b6bfc88" (UID: "286dd146-aa4a-4d43-9324-eae87b6bfc88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:37:52.630983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.630941 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6rp\" (UniqueName: \"kubernetes.io/projected/286dd146-aa4a-4d43-9324-eae87b6bfc88-kube-api-access-zr6rp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:52.630983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.630982 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:52.630983 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:52.630992 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/286dd146-aa4a-4d43-9324-eae87b6bfc88-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.031462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.031434 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:37:53.136487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136420 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136456 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136489 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136608 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136649 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136691 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136739 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136832 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.136926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136841 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:37:53.136926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136875 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token\") pod \"7c3da7da-e58f-41cc-99ea-03e12f949416\" (UID: \"7c3da7da-e58f-41cc-99ea-03e12f949416\") " Oct 31 23:37:53.137135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.136982 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:37:53.137135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137014 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:37:53.137135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137052 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:37:53.137272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137210 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.137272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137215 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:37:53.137272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137234 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3da7da-e58f-41cc-99ea-03e12f949416-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.137272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137250 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.137272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.137263 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.138753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.138728 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:37:53.138859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.138839 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:37:53.138859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.138850 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "kube-api-access-glbgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:37:53.138978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.138957 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo" pod "7c3da7da-e58f-41cc-99ea-03e12f949416" (UID: "7c3da7da-e58f-41cc-99ea-03e12f949416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:37:53.237975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.237954 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.237975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.237974 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.238112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.237984 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glbgw\" (UniqueName: \"kubernetes.io/projected/7c3da7da-e58f-41cc-99ea-03e12f949416-kube-api-access-glbgw\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.238112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.237994 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.238112 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.238002 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3da7da-e58f-41cc-99ea-03e12f949416-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:37:53.303368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.303347 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" Oct 31 23:37:53.303488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.303384 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c" event={"ID":"286dd146-aa4a-4d43-9324-eae87b6bfc88","Type":"ContainerDied","Data":"7935557fb8ede1a68cc7ea14b73e2890939d88c129923eefc6e5fe6009621350"} Oct 31 23:37:53.303488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.303425 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7935557fb8ede1a68cc7ea14b73e2890939d88c129923eefc6e5fe6009621350" Oct 31 23:37:53.305143 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.305119 5043 generic.go:356] "Generic (PLEG): container finished" podID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerID="90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394" exitCode=0 Oct 31 23:37:53.305244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.305169 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" Oct 31 23:37:53.305244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.305178 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" event={"ID":"7c3da7da-e58f-41cc-99ea-03e12f949416","Type":"ContainerDied","Data":"90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394"} Oct 31 23:37:53.305244 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.305230 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq" event={"ID":"7c3da7da-e58f-41cc-99ea-03e12f949416","Type":"ContainerDied","Data":"eb81bc13f130d97eaea0830b65cada2005654d7619d081d2fcfff19f04d72e14"} Oct 31 23:37:53.305429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.305259 5043 scope.go:122] "RemoveContainer" containerID="90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394" Oct 31 23:37:53.318089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.318070 5043 scope.go:122] "RemoveContainer" containerID="81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab" Oct 31 23:37:53.326354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.326339 5043 scope.go:122] "RemoveContainer" containerID="90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394" Oct 31 23:37:53.326630 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:37:53.326613 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394\": container with ID starting with 90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394 not found: ID does not exist" containerID="90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394" Oct 31 23:37:53.326702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.326637 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394"} err="failed to get container status \"90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394\": rpc error: code = NotFound desc = could not find container \"90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394\": container with ID starting with 90e42d2c9e3bfb39fe02be263a7392a97d1fce683e301ab89feaf1767bb8f394 not found: ID does not exist" Oct 31 23:37:53.326702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.326653 5043 scope.go:122] "RemoveContainer" containerID="81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab" Oct 31 23:37:53.326880 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:37:53.326864 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab\": container with ID starting with 81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab not found: ID does not exist" containerID="81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab" Oct 31 23:37:53.326931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.326886 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab"} err="failed to get container status \"81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab\": rpc error: code = NotFound desc = could not find container \"81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab\": container with ID starting with 81e0b7f95808815e53fc28f1c421ce709bea1a6df059c806ae991cc5cb871aab not found: ID does not exist" Oct 31 23:37:53.335741 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.335719 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq"] Oct 31 23:37:53.339556 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.339525 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c4619f-openshift-default-f9bfcb59c-cfrbq"] Oct 31 23:37:53.934147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:37:53.934111 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" path="/var/lib/kubelet/pods/7c3da7da-e58f-41cc-99ea-03e12f949416/volumes" Oct 31 23:38:07.818070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:07.818034 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8"] Oct 31 23:38:07.818640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:07.818239 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" containerID="cri-o://26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c" gracePeriod=10 Oct 31 23:38:08.080690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.080632 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:38:08.176505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.176472 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d\" (UniqueName: \"kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d\") pod \"e0a050c1-60b2-4ee9-a860-8f646427645e\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " Oct 31 23:38:08.176717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.176571 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume\") pod \"e0a050c1-60b2-4ee9-a860-8f646427645e\" (UID: \"e0a050c1-60b2-4ee9-a860-8f646427645e\") " Oct 31 23:38:08.177008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.176982 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume" pod "e0a050c1-60b2-4ee9-a860-8f646427645e" (UID: "e0a050c1-60b2-4ee9-a860-8f646427645e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:38:08.178473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.178444 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d" pod "e0a050c1-60b2-4ee9-a860-8f646427645e" (UID: "e0a050c1-60b2-4ee9-a860-8f646427645e"). InnerVolumeSpecName "kube-api-access-9967d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:38:08.277657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.277622 5043 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0a050c1-60b2-4ee9-a860-8f646427645e-extensions-socket-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:38:08.277657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.277657 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9967d\" (UniqueName: \"kubernetes.io/projected/e0a050c1-60b2-4ee9-a860-8f646427645e-kube-api-access-9967d\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:38:08.388726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.388581 5043 generic.go:356] "Generic (PLEG): container finished" podID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerID="26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c" exitCode=0 Oct 31 23:38:08.388726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.388670 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" event={"ID":"e0a050c1-60b2-4ee9-a860-8f646427645e","Type":"ContainerDied","Data":"26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c"} Oct 31 23:38:08.388906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.388734 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" event={"ID":"e0a050c1-60b2-4ee9-a860-8f646427645e","Type":"ContainerDied","Data":"fefc23149435e4d991a70cb2f65578924e9d259c4c31496b4ca039a70430355c"} Oct 31 23:38:08.388906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.388763 5043 scope.go:122] "RemoveContainer" containerID="26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c" Oct 31 23:38:08.389605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.389583 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8" Oct 31 23:38:08.400593 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.400577 5043 scope.go:122] "RemoveContainer" containerID="d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916" Oct 31 23:38:08.409892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.409873 5043 scope.go:122] "RemoveContainer" containerID="26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c" Oct 31 23:38:08.410165 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:38:08.410140 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c\": container with ID starting with 26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c not found: ID does not exist" containerID="26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c" Oct 31 23:38:08.410256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.410167 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c"} err="failed to get container status \"26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c\": rpc error: code = NotFound desc = could not find container \"26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c\": container with ID starting with 26c0a4c68c1f399af6fb7b9893d017660775be7d087af631504922c2b667419c not found: ID does not exist" Oct 31 23:38:08.410256 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.410182 5043 scope.go:122] "RemoveContainer" containerID="d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916" Oct 31 23:38:08.410440 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:38:08.410421 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916\": container with ID starting with d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916 not found: ID does not exist" containerID="d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916" Oct 31 23:38:08.410483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.410447 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916"} err="failed to get container status \"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916\": rpc error: code = NotFound desc = could not find container \"d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916\": container with ID starting with d2da3b057d2ebf37a0daa0d6b318d2954c3b2ea147d698ad20ea3a8a4b665916 not found: ID does not exist" Oct 31 23:38:08.414794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.414772 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8"] Oct 31 23:38:08.420734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:08.420712 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xhfq8"] Oct 31 23:38:09.937967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:09.937133 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" path="/var/lib/kubelet/pods/e0a050c1-60b2-4ee9-a860-8f646427645e/volumes" Oct 31 23:38:10.454592 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:38:10.454556 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:38:11.442073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:11.442040 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:38:12.274327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:12.274291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:38:15.306490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:15.306409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:38:17.747961 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:17.747929 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:38:34.797976 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.797933 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j"] Oct 31 23:38:34.798376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798334 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" Oct 31 23:38:34.798376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798350 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798379 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798384 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798392 5043 cpu_manager.go:414] "containerMap: removing container" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798397 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798409 5043 cpu_manager.go:414] "containerMap: removing container" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798413 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798446 5043 cpu_manager.go:414] "containerMap: removing container" podUID="286dd146-aa4a-4d43-9324-eae87b6bfc88" containerName="revoke-keys" Oct 31 23:38:34.798484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798453 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286dd146-aa4a-4d43-9324-eae87b6bfc88" containerName="revoke-keys" Oct 31 23:38:34.798811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798520 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="286dd146-aa4a-4d43-9324-eae87b6bfc88" containerName="revoke-keys" Oct 31 23:38:34.798811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798529 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" Oct 31 23:38:34.798811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798539 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" Oct 31 23:38:34.798811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798556 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3da7da-e58f-41cc-99ea-03e12f949416" containerName="istio-proxy" Oct 31 23:38:34.798811 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.798565 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a050c1-60b2-4ee9-a860-8f646427645e" containerName="manager" Oct 31 23:38:34.801790 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.801770 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.815239 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.815215 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j"] Oct 31 23:38:34.901879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.901852 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.901879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.901884 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.901908 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.901972 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902130 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.902046 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.902128 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52de0ae-1237-4099-90e1-a65f810ea3b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.902175 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.902206 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pw5r\" (UniqueName: \"kubernetes.io/projected/d52de0ae-1237-4099-90e1-a65f810ea3b4-kube-api-access-2pw5r\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:34.902285 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:34.902253 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003045 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003008 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52de0ae-1237-4099-90e1-a65f810ea3b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003060 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003103 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pw5r\" (UniqueName: \"kubernetes.io/projected/d52de0ae-1237-4099-90e1-a65f810ea3b4-kube-api-access-2pw5r\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003220 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003151 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003253 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003283 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003312 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003365 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003406 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003394 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003653 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003728 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003774 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003808 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.003993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.003968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d52de0ae-1237-4099-90e1-a65f810ea3b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.005571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.005520 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.005900 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.005884 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.015995 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.015968 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d52de0ae-1237-4099-90e1-a65f810ea3b4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.016217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.016198 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pw5r\" (UniqueName: \"kubernetes.io/projected/d52de0ae-1237-4099-90e1-a65f810ea3b4-kube-api-access-2pw5r\") pod \"maas-default-gateway-openshift-default-6886b96f9-l992j\" (UID: \"d52de0ae-1237-4099-90e1-a65f810ea3b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.115796 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.115716 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:35.401169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.401144 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j"] Oct 31 23:38:35.403309 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:38:35.403286 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52de0ae_1237_4099_90e1_a65f810ea3b4.slice/crio-51c530db4002871138777a1de4af6c7bc7e5519a0f7e54af7e2935082bef1d5f WatchSource:0}: Error finding container 51c530db4002871138777a1de4af6c7bc7e5519a0f7e54af7e2935082bef1d5f: Status 404 returned error can't find the container with id 51c530db4002871138777a1de4af6c7bc7e5519a0f7e54af7e2935082bef1d5f Oct 31 23:38:35.405775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.405732 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:35.405898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.405820 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:35.405898 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.405855 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:35.543537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.543499 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" event={"ID":"d52de0ae-1237-4099-90e1-a65f810ea3b4","Type":"ContainerStarted","Data":"d5048a2d5eb08e08e3d8700fecf274ded2b6d43a1ebaa13de28cc4d1c65410a0"} Oct 31 23:38:35.543758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.543569 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" event={"ID":"d52de0ae-1237-4099-90e1-a65f810ea3b4","Type":"ContainerStarted","Data":"51c530db4002871138777a1de4af6c7bc7e5519a0f7e54af7e2935082bef1d5f"} Oct 31 23:38:35.569080 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:35.568180 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podStartSLOduration=1.568162698 podStartE2EDuration="1.568162698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:35.566274717 +0000 UTC m=+2941.795829125" watchObservedRunningTime="2026-10-31 23:38:35.568162698 +0000 UTC m=+2941.797717109" Oct 31 23:38:36.116296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:36.116261 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:36.121210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:36.121185 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:36.548291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:36.548262 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:36.549039 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:36.549017 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:38:40.456244 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:38:40.456207 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:38:49.803403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.803369 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc"] Oct 31 23:38:49.808479 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.808452 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.822425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.822393 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc"] Oct 31 23:38:49.833701 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.833679 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4"] Oct 31 23:38:49.837277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.837246 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842314 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842357 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842391 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.842461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842416 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.842461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842446 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578ea20-3979-4843-98cf-78864e3774b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842472 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842496 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6fc\" (UniqueName: \"kubernetes.io/projected/39400b70-1b3e-4c2c-9ab8-a45e349837ad-kube-api-access-rj6fc\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842524 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578ea20-3979-4843-98cf-78864e3774b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842563 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842590 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842624 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxlp\" (UniqueName: \"kubernetes.io/projected/7578ea20-3979-4843-98cf-78864e3774b6-kube-api-access-jgxlp\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842669 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842717 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.842768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842764 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.843269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842789 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.843269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842813 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578ea20-3979-4843-98cf-78864e3774b6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.843269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842844 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.843269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.842863 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.849088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.849065 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4"] Oct 31 23:38:49.943778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943751 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.943973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943785 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.943973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943806 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.943973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943827 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.943973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943847 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578ea20-3979-4843-98cf-78864e3774b6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.943973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943878 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.943973 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.943895 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.944537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944141 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.944537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944183 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.944537 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944218 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944242 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944659 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944678 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578ea20-3979-4843-98cf-78864e3774b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944623 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944714 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944739 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6fc\" (UniqueName: \"kubernetes.io/projected/39400b70-1b3e-4c2c-9ab8-a45e349837ad-kube-api-access-rj6fc\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.944775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944766 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578ea20-3979-4843-98cf-78864e3774b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944789 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944816 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944847 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxlp\" (UniqueName: \"kubernetes.io/projected/7578ea20-3979-4843-98cf-78864e3774b6-kube-api-access-jgxlp\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944952 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944967 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.944375 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.945154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.945038 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.945516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.945176 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.945516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.945235 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.945516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.945356 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.945700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.945645 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578ea20-3979-4843-98cf-78864e3774b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.946681 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.946626 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578ea20-3979-4843-98cf-78864e3774b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.946789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.946746 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.946831 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.946805 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.947270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.947255 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578ea20-3979-4843-98cf-78864e3774b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.956466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.956357 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578ea20-3979-4843-98cf-78864e3774b6-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.956828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.956807 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxlp\" (UniqueName: \"kubernetes.io/projected/7578ea20-3979-4843-98cf-78864e3774b6-kube-api-access-jgxlp\") pod \"maas-default-gateway-openshift-default-6886b96f9-smff4\" (UID: \"7578ea20-3979-4843-98cf-78864e3774b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:49.957464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.957243 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39400b70-1b3e-4c2c-9ab8-a45e349837ad-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:49.957464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:49.957452 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6fc\" (UniqueName: \"kubernetes.io/projected/39400b70-1b3e-4c2c-9ab8-a45e349837ad-kube-api-access-rj6fc\") pod \"maas-default-gateway-openshift-default-6886b96f9-62pjc\" (UID: \"39400b70-1b3e-4c2c-9ab8-a45e349837ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:50.122922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.122858 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:50.154327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.154303 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:50.406727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.406699 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc"] Oct 31 23:38:50.408304 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:38:50.408276 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39400b70_1b3e_4c2c_9ab8_a45e349837ad.slice/crio-ee77fc0beda8ced9770e62f383e53674fd86abfcab1925b936fc9690e3e9f1b8 WatchSource:0}: Error finding container ee77fc0beda8ced9770e62f383e53674fd86abfcab1925b936fc9690e3e9f1b8: Status 404 returned error can't find the container with id ee77fc0beda8ced9770e62f383e53674fd86abfcab1925b936fc9690e3e9f1b8 Oct 31 23:38:50.410430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.410400 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:50.410502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.410461 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:50.410502 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.410488 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:50.448700 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.448679 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4"] Oct 31 23:38:50.450793 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:38:50.450769 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7578ea20_3979_4843_98cf_78864e3774b6.slice/crio-4e646be74adfd7270a4e684418fabbfd448b5ec3bfaaf9bf2cf84f3854527a88 WatchSource:0}: Error finding container 4e646be74adfd7270a4e684418fabbfd448b5ec3bfaaf9bf2cf84f3854527a88: Status 404 returned error can't find the container with id 4e646be74adfd7270a4e684418fabbfd448b5ec3bfaaf9bf2cf84f3854527a88 Oct 31 23:38:50.453157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.453130 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:50.453255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.453191 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:50.453255 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.453222 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:38:50.628666 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.628614 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" event={"ID":"39400b70-1b3e-4c2c-9ab8-a45e349837ad","Type":"ContainerStarted","Data":"255cd5f21fc51a389533a8bdbdfbf1d51c3c82fde63be315715d6c2663865fbf"} Oct 31 23:38:50.628851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.628704 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" event={"ID":"39400b70-1b3e-4c2c-9ab8-a45e349837ad","Type":"ContainerStarted","Data":"ee77fc0beda8ced9770e62f383e53674fd86abfcab1925b936fc9690e3e9f1b8"} Oct 31 23:38:50.630509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.630478 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" event={"ID":"7578ea20-3979-4843-98cf-78864e3774b6","Type":"ContainerStarted","Data":"12518883df6097f44f1eccc6c2b0e053779e900618de6460fad9d64ff63ae5b7"} Oct 31 23:38:50.630706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.630567 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" event={"ID":"7578ea20-3979-4843-98cf-78864e3774b6","Type":"ContainerStarted","Data":"4e646be74adfd7270a4e684418fabbfd448b5ec3bfaaf9bf2cf84f3854527a88"} Oct 31 23:38:50.674446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.674332 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podStartSLOduration=1.674316132 podStartE2EDuration="1.674316132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:50.673715123 +0000 UTC m=+2956.903269547" watchObservedRunningTime="2026-10-31 23:38:50.674316132 +0000 UTC m=+2956.903870542" Oct 31 23:38:50.675708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:50.675676 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podStartSLOduration=1.6756666249999999 podStartE2EDuration="1.675666625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:38:50.651045947 +0000 UTC m=+2956.880600364" watchObservedRunningTime="2026-10-31 23:38:50.675666625 +0000 UTC m=+2956.905221035" Oct 31 23:38:51.123326 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:51.123291 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:51.124645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:51.124620 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:38:51.124765 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:51.124692 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:38:51.154615 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:51.154590 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:51.155628 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:51.155610 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:38:51.155679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:51.155663 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:38:52.127909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:52.127878 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:52.158906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:52.158878 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:52.640818 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:52.640777 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:52.641034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:52.640836 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:52.641649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:52.641627 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:38:52.641770 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:52.641696 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:38:55.011917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:38:55.011884 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:39:10.455650 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:39:10.455612 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:39:15.366610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:15.366579 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:39:18.053486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:18.053452 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:39:18.194290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:18.194258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:39:26.581153 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.581119 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr"] Oct 31 23:39:26.585250 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.585231 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.593760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.593734 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tgrvp\"" Oct 31 23:39:26.596465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.596439 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr"] Oct 31 23:39:26.675303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.675270 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq62j\" (UniqueName: \"kubernetes.io/projected/cd2d832b-995b-41b4-b9c5-8372b2ec54a5-kube-api-access-tq62j\") pod \"kuadrant-operator-controller-manager-678d87766f-xdnsr\" (UID: \"cd2d832b-995b-41b4-b9c5-8372b2ec54a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.675446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.675342 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd2d832b-995b-41b4-b9c5-8372b2ec54a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xdnsr\" (UID: \"cd2d832b-995b-41b4-b9c5-8372b2ec54a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.776015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.775986 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq62j\" (UniqueName: \"kubernetes.io/projected/cd2d832b-995b-41b4-b9c5-8372b2ec54a5-kube-api-access-tq62j\") pod \"kuadrant-operator-controller-manager-678d87766f-xdnsr\" (UID: \"cd2d832b-995b-41b4-b9c5-8372b2ec54a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.776156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.776075 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd2d832b-995b-41b4-b9c5-8372b2ec54a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xdnsr\" (UID: \"cd2d832b-995b-41b4-b9c5-8372b2ec54a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.776439 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.776421 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd2d832b-995b-41b4-b9c5-8372b2ec54a5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xdnsr\" (UID: \"cd2d832b-995b-41b4-b9c5-8372b2ec54a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.788477 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.788453 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq62j\" (UniqueName: \"kubernetes.io/projected/cd2d832b-995b-41b4-b9c5-8372b2ec54a5-kube-api-access-tq62j\") pod \"kuadrant-operator-controller-manager-678d87766f-xdnsr\" (UID: \"cd2d832b-995b-41b4-b9c5-8372b2ec54a5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:26.899842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:26.899765 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:27.190422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:27.190352 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr"] Oct 31 23:39:27.194801 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:39:27.194773 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2d832b_995b_41b4_b9c5_8372b2ec54a5.slice/crio-90819896081e20afd1c1c435a583c41f6197b1ba7c403d8d263bf3244ab35f07 WatchSource:0}: Error finding container 90819896081e20afd1c1c435a583c41f6197b1ba7c403d8d263bf3244ab35f07: Status 404 returned error can't find the container with id 90819896081e20afd1c1c435a583c41f6197b1ba7c403d8d263bf3244ab35f07 Oct 31 23:39:27.824038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:27.824005 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" event={"ID":"cd2d832b-995b-41b4-b9c5-8372b2ec54a5","Type":"ContainerStarted","Data":"d25baec4252cb1fdc1f432f552e992b9790456763d9aa44b05cb5b0aa657292f"} Oct 31 23:39:27.824461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:27.824060 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:27.824461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:27.824077 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" event={"ID":"cd2d832b-995b-41b4-b9c5-8372b2ec54a5","Type":"ContainerStarted","Data":"90819896081e20afd1c1c435a583c41f6197b1ba7c403d8d263bf3244ab35f07"} Oct 31 23:39:27.850580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:27.850520 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" podStartSLOduration=1.850507898 podStartE2EDuration="1.850507898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:39:27.847245454 +0000 UTC m=+2994.076799863" watchObservedRunningTime="2026-10-31 23:39:27.850507898 +0000 UTC m=+2994.080062305" Oct 31 23:39:38.832373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:38.832338 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xdnsr" Oct 31 23:39:40.455484 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:39:40.455447 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:39:53.682767 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:53.682702 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:39:53.683336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:53.682817 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:39:53.683715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:53.683682 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:39:53.683859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:53.683743 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:39:58.103272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.103240 5043 generic.go:356] "Generic (PLEG): container finished" podID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerID="d5048a2d5eb08e08e3d8700fecf274ded2b6d43a1ebaa13de28cc4d1c65410a0" exitCode=0 Oct 31 23:39:58.103797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.103311 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" event={"ID":"d52de0ae-1237-4099-90e1-a65f810ea3b4","Type":"ContainerDied","Data":"d5048a2d5eb08e08e3d8700fecf274ded2b6d43a1ebaa13de28cc4d1c65410a0"} Oct 31 23:39:58.103797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.103716 5043 scope.go:122] "RemoveContainer" containerID="d5048a2d5eb08e08e3d8700fecf274ded2b6d43a1ebaa13de28cc4d1c65410a0" Oct 31 23:39:58.104940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.104906 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.105022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.104985 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.105022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.104996 5043 generic.go:356] "Generic (PLEG): container finished" podID="7578ea20-3979-4843-98cf-78864e3774b6" containerID="12518883df6097f44f1eccc6c2b0e053779e900618de6460fad9d64ff63ae5b7" exitCode=0 Oct 31 23:39:58.105022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.105015 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.105124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.105066 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" event={"ID":"7578ea20-3979-4843-98cf-78864e3774b6","Type":"ContainerDied","Data":"12518883df6097f44f1eccc6c2b0e053779e900618de6460fad9d64ff63ae5b7"} Oct 31 23:39:58.105367 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.105350 5043 scope.go:122] "RemoveContainer" containerID="12518883df6097f44f1eccc6c2b0e053779e900618de6460fad9d64ff63ae5b7" Oct 31 23:39:58.106363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.106334 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.106462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.106405 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.106462 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.106444 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.106997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.106897 5043 generic.go:356] "Generic (PLEG): container finished" podID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerID="255cd5f21fc51a389533a8bdbdfbf1d51c3c82fde63be315715d6c2663865fbf" exitCode=0 Oct 31 23:39:58.106997 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.106932 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" event={"ID":"39400b70-1b3e-4c2c-9ab8-a45e349837ad","Type":"ContainerDied","Data":"255cd5f21fc51a389533a8bdbdfbf1d51c3c82fde63be315715d6c2663865fbf"} Oct 31 23:39:58.107234 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.107204 5043 scope.go:122] "RemoveContainer" containerID="255cd5f21fc51a389533a8bdbdfbf1d51c3c82fde63be315715d6c2663865fbf" Oct 31 23:39:58.108304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.108214 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.108304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.108284 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.108413 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.108324 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:39:58.795526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.795488 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp"] Oct 31 23:39:58.799056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.799037 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:58.803427 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.803072 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:39:58.805005 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.804972 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:39:58.805298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.805282 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:39:58.805527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.805512 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:39:58.806429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.806408 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp"] Oct 31 23:39:58.902293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.902193 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgg7x\" (UniqueName: \"kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:58.902293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.902252 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:58.902584 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:58.902438 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.003877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.003306 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.003877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.003339 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgg7x\" (UniqueName: \"kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.003877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.003365 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.004269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.004241 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.005960 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.005932 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.049620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.049496 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgg7x\" (UniqueName: \"kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x\") pod \"maas-api-revoke-sub-d576e68b-p7ccp\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.133570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.118274 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" event={"ID":"39400b70-1b3e-4c2c-9ab8-a45e349837ad","Type":"ContainerStarted","Data":"8422b08392be1ea95d31ee1cf9c8ba805dadce15f07fb790f8d71ee5e972339a"} Oct 31 23:39:59.145571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.140331 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" event={"ID":"d52de0ae-1237-4099-90e1-a65f810ea3b4","Type":"ContainerStarted","Data":"dddafffd612ba29938f9245e4488254525cea3176495373ecc143ce544af6a7e"} Oct 31 23:39:59.145571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.141712 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:39:59.149563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.146615 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:39:59.149563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.146965 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:39:59.149563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.147059 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:39:59.149563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.147113 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:39:59.171575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.167683 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:39:59.171575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.167750 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:39:59.199691 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.199650 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" event={"ID":"7578ea20-3979-4843-98cf-78864e3774b6","Type":"ContainerStarted","Data":"2151a153f72689d04d275cf1c081cd1278f8954617c9e972db9652bdd12dcbdf"} Oct 31 23:39:59.591433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:39:59.591332 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp"] Oct 31 23:39:59.600461 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:39:59.600420 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204c09bf_29f4_459e_8982_ced30c3e87b1.slice/crio-1b0cfbfd473f1aef713602a8e90911ad5f420a6c3c975378da5a3ac8313e61fc WatchSource:0}: Error finding container 1b0cfbfd473f1aef713602a8e90911ad5f420a6c3c975378da5a3ac8313e61fc: Status 404 returned error can't find the container with id 1b0cfbfd473f1aef713602a8e90911ad5f420a6c3c975378da5a3ac8313e61fc Oct 31 23:40:00.118345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.117906 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:40:00.118345 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.117966 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:40:00.127459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.127337 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:40:00.127459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.127406 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:40:00.162017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.161991 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:40:00.162935 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.162910 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:40:00.163118 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.163094 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:40:00.207446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.207417 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" event={"ID":"204c09bf-29f4-459e-8982-ced30c3e87b1","Type":"ContainerStarted","Data":"9eac086b493425f591d040d7ac51a734a6efa3d13a3be7ae59a3c9430844eb9d"} Oct 31 23:40:00.207611 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.207463 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" event={"ID":"204c09bf-29f4-459e-8982-ced30c3e87b1","Type":"ContainerStarted","Data":"1b0cfbfd473f1aef713602a8e90911ad5f420a6c3c975378da5a3ac8313e61fc"} Oct 31 23:40:00.236620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:00.235666 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" podStartSLOduration=2.235647978 podStartE2EDuration="2.235647978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:40:00.233840069 +0000 UTC m=+3026.463394505" watchObservedRunningTime="2026-10-31 23:40:00.235647978 +0000 UTC m=+3026.465202389" Oct 31 23:40:01.118093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.117993 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:40:01.118682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.118639 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:40:01.129484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.129449 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:40:01.129772 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.129743 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:40:01.169788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.169749 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:40:01.170437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.170391 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:40:01.213121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.213090 5043 generic.go:356] "Generic (PLEG): container finished" podID="204c09bf-29f4-459e-8982-ced30c3e87b1" containerID="9eac086b493425f591d040d7ac51a734a6efa3d13a3be7ae59a3c9430844eb9d" exitCode=0 Oct 31 23:40:01.213304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:01.213179 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" event={"ID":"204c09bf-29f4-459e-8982-ced30c3e87b1","Type":"ContainerDied","Data":"9eac086b493425f591d040d7ac51a734a6efa3d13a3be7ae59a3c9430844eb9d"} Oct 31 23:40:02.122074 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.122038 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:40:02.122392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.122355 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:40:02.123277 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.123241 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:40:02.123463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.123443 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:40:02.164487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.164447 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:40:02.164814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.164781 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:40:02.688904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.688814 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:40:02.695465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.695437 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca\") pod \"204c09bf-29f4-459e-8982-ced30c3e87b1\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " Oct 31 23:40:02.695638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.695569 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca\") pod \"204c09bf-29f4-459e-8982-ced30c3e87b1\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " Oct 31 23:40:02.695638 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.695601 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x\" (UniqueName: \"kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x\") pod \"204c09bf-29f4-459e-8982-ced30c3e87b1\" (UID: \"204c09bf-29f4-459e-8982-ced30c3e87b1\") " Oct 31 23:40:02.696085 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.696064 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca" pod "204c09bf-29f4-459e-8982-ced30c3e87b1" (UID: "204c09bf-29f4-459e-8982-ced30c3e87b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:40:02.697969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.697946 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x" pod "204c09bf-29f4-459e-8982-ced30c3e87b1" (UID: "204c09bf-29f4-459e-8982-ced30c3e87b1"). InnerVolumeSpecName "kube-api-access-rgg7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:40:02.698175 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.698162 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca" pod "204c09bf-29f4-459e-8982-ced30c3e87b1" (UID: "204c09bf-29f4-459e-8982-ced30c3e87b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:40:02.796518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.796477 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgg7x\" (UniqueName: \"kubernetes.io/projected/204c09bf-29f4-459e-8982-ced30c3e87b1-kube-api-access-rgg7x\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:40:02.796518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.796511 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:40:02.796518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:02.796523 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204c09bf-29f4-459e-8982-ced30c3e87b1-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:40:03.118015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.117973 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:40:03.118314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.118286 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:40:03.123925 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.123894 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:40:03.124164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.124133 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:40:03.156214 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.156179 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:40:03.156499 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.156471 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:40:03.275581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.273304 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" event={"ID":"204c09bf-29f4-459e-8982-ced30c3e87b1","Type":"ContainerDied","Data":"1b0cfbfd473f1aef713602a8e90911ad5f420a6c3c975378da5a3ac8313e61fc"} Oct 31 23:40:03.275581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.273372 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0cfbfd473f1aef713602a8e90911ad5f420a6c3c975378da5a3ac8313e61fc" Oct 31 23:40:03.275581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:03.273439 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp" Oct 31 23:40:04.116320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:04.116280 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:40:04.116756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:04.116340 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:40:04.123379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:04.123347 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:40:04.123581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:04.123406 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:40:04.154951 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:04.154906 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:40:04.155138 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:04.154974 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:40:05.116209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:05.116171 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l992j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 23:40:05.116422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:05.116223 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" podUID="d52de0ae-1237-4099-90e1-a65f810ea3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 23:40:05.123557 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:05.123511 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-62pjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 23:40:05.123682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:05.123580 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" podUID="39400b70-1b3e-4c2c-9ab8-a45e349837ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 23:40:05.155539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:05.155508 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body= Oct 31 23:40:05.155677 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:05.155581 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" Oct 31 23:40:06.548608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.548568 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:40:06.656505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.656477 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:40:06.657839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.657814 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:40:06.658599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.658581 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-62pjc" Oct 31 23:40:06.758598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.758540 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:40:06.852464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.852388 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:40:06.853494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.853469 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:40:06.854291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:06.854270 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" Oct 31 23:40:07.303660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:07.303625 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l992j" Oct 31 23:40:10.458222 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:40:10.458176 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:40:15.547949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:15.547901 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:17.850896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.850854 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn"] Oct 31 23:40:17.851421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.851306 5043 cpu_manager.go:414] "containerMap: removing container" podUID="204c09bf-29f4-459e-8982-ced30c3e87b1" containerName="revoke-keys" Oct 31 23:40:17.851421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.851318 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204c09bf-29f4-459e-8982-ced30c3e87b1" containerName="revoke-keys" Oct 31 23:40:17.851421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.851406 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="204c09bf-29f4-459e-8982-ced30c3e87b1" containerName="revoke-keys" Oct 31 23:40:17.854778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.854756 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:17.859096 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.859068 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:40:17.861478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.861451 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:40:17.861797 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.861775 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:40:17.862305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.862285 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:40:17.863975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.863952 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn"] Oct 31 23:40:17.983757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.983730 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5lb\" (UniqueName: \"kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:17.983919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.983761 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:17.983919 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:17.983791 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.084577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.084534 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5lb\" (UniqueName: \"kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.084751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.084580 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.084751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.084719 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.085223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.085201 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.087058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.087041 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.096912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.096892 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5lb\" (UniqueName: \"kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb\") pod \"maas-api-revoke-sub-d2bc93bc-sh9cn\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.171504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.171453 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:18.447579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.447532 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn"] Oct 31 23:40:18.449588 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:40:18.449557 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ed33c1_9bf1_4018_bbd6_547f64dd84ad.slice/crio-664b77c70116f689f134b4bf08ba655c2f6e7eabe474a83a9a1191b2f7f4fd3f WatchSource:0}: Error finding container 664b77c70116f689f134b4bf08ba655c2f6e7eabe474a83a9a1191b2f7f4fd3f: Status 404 returned error can't find the container with id 664b77c70116f689f134b4bf08ba655c2f6e7eabe474a83a9a1191b2f7f4fd3f Oct 31 23:40:18.475163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:18.475137 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" event={"ID":"49ed33c1-9bf1-4018-bbd6-547f64dd84ad","Type":"ContainerStarted","Data":"664b77c70116f689f134b4bf08ba655c2f6e7eabe474a83a9a1191b2f7f4fd3f"} Oct 31 23:40:19.482902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:19.482872 5043 generic.go:356] "Generic (PLEG): container finished" podID="49ed33c1-9bf1-4018-bbd6-547f64dd84ad" containerID="5701a25140ef4e3fed2122665e641fc73ed3f92a337ddde66dcb7aaf6b9a69b9" exitCode=0 Oct 31 23:40:19.483310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:19.482961 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" event={"ID":"49ed33c1-9bf1-4018-bbd6-547f64dd84ad","Type":"ContainerDied","Data":"5701a25140ef4e3fed2122665e641fc73ed3f92a337ddde66dcb7aaf6b9a69b9"} Oct 31 23:40:20.634837 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.634812 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:20.809823 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.809784 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca\") pod \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " Oct 31 23:40:20.809986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.809892 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca\") pod \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " Oct 31 23:40:20.809986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.809958 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb\" (UniqueName: \"kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb\") pod \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\" (UID: \"49ed33c1-9bf1-4018-bbd6-547f64dd84ad\") " Oct 31 23:40:20.810297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.810263 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca" pod "49ed33c1-9bf1-4018-bbd6-547f64dd84ad" (UID: "49ed33c1-9bf1-4018-bbd6-547f64dd84ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:40:20.812050 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.812026 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca" pod "49ed33c1-9bf1-4018-bbd6-547f64dd84ad" (UID: "49ed33c1-9bf1-4018-bbd6-547f64dd84ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:40:20.812127 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.812081 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb" pod "49ed33c1-9bf1-4018-bbd6-547f64dd84ad" (UID: "49ed33c1-9bf1-4018-bbd6-547f64dd84ad"). InnerVolumeSpecName "kube-api-access-9x5lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:40:20.910639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.910611 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:40:20.910639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.910635 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5lb\" (UniqueName: \"kubernetes.io/projected/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-kube-api-access-9x5lb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:40:20.910817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:20.910645 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ed33c1-9bf1-4018-bbd6-547f64dd84ad-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:40:21.494273 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:21.494242 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" Oct 31 23:40:21.494460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:21.494280 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn" event={"ID":"49ed33c1-9bf1-4018-bbd6-547f64dd84ad","Type":"ContainerDied","Data":"664b77c70116f689f134b4bf08ba655c2f6e7eabe474a83a9a1191b2f7f4fd3f"} Oct 31 23:40:21.494460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:21.494356 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="664b77c70116f689f134b4bf08ba655c2f6e7eabe474a83a9a1191b2f7f4fd3f" Oct 31 23:40:29.573748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.573709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:29.899066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.898963 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4"] Oct 31 23:40:29.899643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.899622 5043 cpu_manager.go:414] "containerMap: removing container" podUID="49ed33c1-9bf1-4018-bbd6-547f64dd84ad" containerName="revoke-keys" Oct 31 23:40:29.899643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.899642 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ed33c1-9bf1-4018-bbd6-547f64dd84ad" containerName="revoke-keys" Oct 31 23:40:29.899833 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.899758 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ed33c1-9bf1-4018-bbd6-547f64dd84ad" containerName="revoke-keys" Oct 31 23:40:29.903635 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.903614 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:29.916093 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.916064 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-abd9b5-openshift-default-dockercfg-ts74d\"" Oct 31 23:40:29.920403 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.920376 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4"] Oct 31 23:40:29.973927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:29.973894 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:30.001446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.001396 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.001774 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.001749 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.001981 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.001946 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.002213 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.002180 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.002527 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.002495 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.003154 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.003132 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.003396 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.003378 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.004317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.004295 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9h8w\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.005626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.005606 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.107720 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107686 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.107927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107736 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9h8w\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.107927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107774 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.107927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107825 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.107927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107847 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.107927 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107872 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107958 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.107988 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108189 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.108108 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.108309 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.108333 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108515 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.108492 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.108589 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.108746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.108705 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.110369 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.110347 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.110464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.110450 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.120062 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.120041 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.120394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.120370 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9h8w\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w\") pod \"e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.197789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.197678 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:30.223944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.223899 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:30.511869 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.511840 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4"] Oct 31 23:40:30.513017 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:40:30.512993 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a68ce20_595c_436d_8cd1_7a4d2505f01d.slice/crio-83ff1483dd4e58c3d0062ab671cea2608a172e1ad7fc34b4542a4982536dc067 WatchSource:0}: Error finding container 83ff1483dd4e58c3d0062ab671cea2608a172e1ad7fc34b4542a4982536dc067: Status 404 returned error can't find the container with id 83ff1483dd4e58c3d0062ab671cea2608a172e1ad7fc34b4542a4982536dc067 Oct 31 23:40:30.515099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.515061 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:40:30.515168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.515138 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:40:30.515209 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.515189 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:40:30.543373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:30.543348 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" event={"ID":"3a68ce20-595c-436d-8cd1-7a4d2505f01d","Type":"ContainerStarted","Data":"83ff1483dd4e58c3d0062ab671cea2608a172e1ad7fc34b4542a4982536dc067"} Oct 31 23:40:31.551786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:31.551749 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" event={"ID":"3a68ce20-595c-436d-8cd1-7a4d2505f01d","Type":"ContainerStarted","Data":"6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e"} Oct 31 23:40:31.580060 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:31.580012 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podStartSLOduration=2.57999915 podStartE2EDuration="2.57999915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:40:31.57730983 +0000 UTC m=+3057.806864240" watchObservedRunningTime="2026-10-31 23:40:31.57999915 +0000 UTC m=+3057.809553559" Oct 31 23:40:32.224979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:32.224948 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:32.230188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:32.230162 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:32.556867 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:32.556831 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:32.557949 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:32.557927 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:33.865106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:33.865061 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:34.005321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:34.005291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:35.028641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.028599 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6"] Oct 31 23:40:35.032324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.032304 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.040394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.040366 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-abd9b5\"" Oct 31 23:40:35.046459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.046431 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:35.046715 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.046683 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6"] Oct 31 23:40:35.099516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.099483 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f"] Oct 31 23:40:35.103163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.103146 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.110450 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.110420 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6e3af9ac-dockercfg-94jkp\"" Oct 31 23:40:35.113609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.113587 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f"] Oct 31 23:40:35.161817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.161783 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbl5\" (UniqueName: \"kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5\") pod \"payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.161992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.161836 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnvp\" (UniqueName: \"kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp\") pod \"payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.161992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.161957 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.162125 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.162107 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.240188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.233115 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:35.263365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.263330 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.263645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.263396 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbl5\" (UniqueName: \"kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5\") pod \"payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.263645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.263432 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnvp\" (UniqueName: \"kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp\") pod \"payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.263645 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.263481 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.264107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.264087 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.264229 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.264206 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.277381 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.277355 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnvp\" (UniqueName: \"kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp\") pod \"payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.277820 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.277796 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbl5\" (UniqueName: \"kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5\") pod \"payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.361471 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.361414 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:35.372608 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.372586 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:35.417308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.417165 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:35.648717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.648633 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6"] Oct 31 23:40:35.658830 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:40:35.658792 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeba8c72_4fe4_45e5_a5a9_f63d74cdecd1.slice/crio-e78d77c933f44dbc67a0a6949557a7fbd9ef98b874cef37161e94947f783df06 WatchSource:0}: Error finding container e78d77c933f44dbc67a0a6949557a7fbd9ef98b874cef37161e94947f783df06: Status 404 returned error can't find the container with id e78d77c933f44dbc67a0a6949557a7fbd9ef98b874cef37161e94947f783df06 Oct 31 23:40:35.732325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.732301 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f"] Oct 31 23:40:35.734095 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:40:35.734069 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad3fc5d_f250_48e9_864d_b84466de0a6d.slice/crio-6f9befe63a1b7e6849eece9296f42533a1b765c23a9466182a5f94b1faa01a68 WatchSource:0}: Error finding container 6f9befe63a1b7e6849eece9296f42533a1b765c23a9466182a5f94b1faa01a68: Status 404 returned error can't find the container with id 6f9befe63a1b7e6849eece9296f42533a1b765c23a9466182a5f94b1faa01a68 Oct 31 23:40:35.820221 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:35.820184 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:36.581575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.581512 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" event={"ID":"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1","Type":"ContainerStarted","Data":"20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18"} Oct 31 23:40:36.582025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.581613 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:36.582025 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.581645 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" event={"ID":"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1","Type":"ContainerStarted","Data":"e78d77c933f44dbc67a0a6949557a7fbd9ef98b874cef37161e94947f783df06"} Oct 31 23:40:36.583286 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.583260 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" event={"ID":"0ad3fc5d-f250-48e9-864d-b84466de0a6d","Type":"ContainerStarted","Data":"29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701"} Oct 31 23:40:36.583431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.583299 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" event={"ID":"0ad3fc5d-f250-48e9-864d-b84466de0a6d","Type":"ContainerStarted","Data":"6f9befe63a1b7e6849eece9296f42533a1b765c23a9466182a5f94b1faa01a68"} Oct 31 23:40:36.583431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.583384 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:36.601336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.601309 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:36.601524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.601489 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" podStartSLOduration=1.601477152 podStartE2EDuration="1.601477152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:40:36.599472045 +0000 UTC m=+3062.829026456" watchObservedRunningTime="2026-10-31 23:40:36.601477152 +0000 UTC m=+3062.831031562" Oct 31 23:40:36.622022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:36.621984 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" podStartSLOduration=1.6219732279999999 podStartE2EDuration="1.621973228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:40:36.620264864 +0000 UTC m=+3062.849819273" watchObservedRunningTime="2026-10-31 23:40:36.621973228 +0000 UTC m=+3062.851527638" Oct 31 23:40:37.314446 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:37.314391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:37.935495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:37.935457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:38.610480 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:38.610444 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:39.235626 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:39.235516 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:39.825532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:39.825463 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:40.402517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:40.402473 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:40.479312 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:40:40.479275 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:40:41.113709 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:41.113656 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:41.840330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:41.840273 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:42.410259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:42.410215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:43.001697 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:43.001662 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:43.676689 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:43.676614 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:44.316858 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:44.316822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:44.915243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:44.915059 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:45.522354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:45.522288 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:46.107897 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.106534 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:46.669600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.669498 5043 generic.go:356] "Generic (PLEG): container finished" podID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerID="6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e" exitCode=0 Oct 31 23:40:46.669600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.669589 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" event={"ID":"3a68ce20-595c-436d-8cd1-7a4d2505f01d","Type":"ContainerDied","Data":"6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e"} Oct 31 23:40:46.670102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.669950 5043 scope.go:122] "RemoveContainer" containerID="6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e" Oct 31 23:40:46.671126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.671100 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:40:46.671178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.671166 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:40:46.671217 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.671194 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:40:46.717658 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:46.717455 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:47.305284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:47.305249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:47.556923 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:47.556783 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:47.593748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:47.593719 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:40:47.593926 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:47.593801 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:40:47.677975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:47.677943 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" event={"ID":"3a68ce20-595c-436d-8cd1-7a4d2505f01d","Type":"ContainerStarted","Data":"9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d"} Oct 31 23:40:47.920194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:47.920072 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:48.235627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:48.233801 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:48.238743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:48.238709 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 23:40:48.238890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:48.238779 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 23:40:48.623208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:48.623177 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:48.906019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:48.905935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:49.134567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.128165 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:49.235625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.234956 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 23:40:49.235625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.235018 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 23:40:49.520567 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.517005 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:49.652208 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.652172 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:49.788303 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.788214 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:49.948726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:49.948655 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:50.233619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:50.232873 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 23:40:50.233619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:50.232938 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 23:40:50.609950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:50.609903 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:51.230572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:51.228924 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 23:40:51.230572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:51.229185 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 23:40:51.236566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:51.231371 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:51.880732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:51.880691 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:52.234073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:52.233977 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" start-of-body= Oct 31 23:40:52.234695 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:52.234638 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.2:15021/healthz/ready\": dial tcp 10.132.1.2:15021: connect: connection refused" Oct 31 23:40:52.507980 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:52.507935 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:53.104109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:53.104072 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:53.229309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:53.229279 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:53.229845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:53.229828 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:53.230591 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:53.230572 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:40:53.714970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:53.714937 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:53.858336 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:53.858300 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:54.517921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:54.517889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:55.219710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:55.219665 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:55.942686 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:55.942651 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:56.529243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:56.529206 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:57.122121 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:57.122071 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:57.714349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:57.714315 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:58.331012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:58.330963 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:59.006129 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:59.006088 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:40:59.608617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:40:59.608586 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:00.211247 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:00.211210 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:00.819388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:00.819358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:01.414583 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:01.414534 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:02.014904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:02.014838 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:02.622373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:02.621861 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:03.215378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:03.215332 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:03.921176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:03.921137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:04.505528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:04.505494 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:05.100107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:05.100073 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:05.896513 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:05.896482 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:06.512577 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:06.512526 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:07.121226 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:07.121187 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:07.720966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:07.720920 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:08.363379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:08.363347 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:09.110444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:09.110408 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:09.697491 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:09.697464 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:10.317975 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:10.317925 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:10.459581 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:41:10.459531 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:41:10.968812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:10.968775 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:11.609706 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:11.609660 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:12.203466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:12.203435 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:12.909452 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:12.909394 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:13.515009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:13.514962 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:14.112168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:14.112134 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:14.705215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:14.705185 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:15.313302 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:15.313264 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:15.593266 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:15.593190 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:15.909269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:15.909170 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:16.519183 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:16.519150 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:17.218137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:17.218078 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:17.745891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:17.745865 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:41:17.751137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:17.751114 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:41:17.760324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:17.760304 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:41:17.866440 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:17.866406 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:19.068751 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:19.068727 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:41:19.073616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:19.073595 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:41:19.082881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:19.082863 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:41:19.529985 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:19.529956 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:19.769292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:19.769225 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:20.435368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:20.435289 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:21.082032 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.082001 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:21.173425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.173396 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj"] Oct 31 23:41:21.177746 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.177725 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.182055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.182033 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:41:21.183319 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.183290 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:41:21.185356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.185333 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj"] Oct 31 23:41:21.242386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.242358 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:41:21.242535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.242510 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:41:21.276999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.276965 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmbb\" (UniqueName: \"kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.277139 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.277006 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.277191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.277135 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.378356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.378261 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmbb\" (UniqueName: \"kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.378356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.378299 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.378356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.378333 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.379191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.379163 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.380739 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.380713 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.390748 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.390725 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmbb\" (UniqueName: \"kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb\") pod \"maas-api-revoke-sub-92910a18-xhwmj\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.495970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.495938 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:21.627298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.626182 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:21.779140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.779117 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj"] Oct 31 23:41:21.781609 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:41:21.781580 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69beb65f_fee0_4fcd_a000_a36b6f45ac4b.slice/crio-4c989d577ad2871c3490621211b8b35871681b38b401d4e19c7127e073f4feb4 WatchSource:0}: Error finding container 4c989d577ad2871c3490621211b8b35871681b38b401d4e19c7127e073f4feb4: Status 404 returned error can't find the container with id 4c989d577ad2871c3490621211b8b35871681b38b401d4e19c7127e073f4feb4 Oct 31 23:41:21.883248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:21.883215 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" event={"ID":"69beb65f-fee0-4fcd-a000-a36b6f45ac4b","Type":"ContainerStarted","Data":"4c989d577ad2871c3490621211b8b35871681b38b401d4e19c7127e073f4feb4"} Oct 31 23:41:22.303402 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:22.303278 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:22.894181 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:22.894140 5043 generic.go:356] "Generic (PLEG): container finished" podID="69beb65f-fee0-4fcd-a000-a36b6f45ac4b" containerID="017b50046e2f293e1dd0916df4908d9292a3a136d9ff4519f2d4660460d7c415" exitCode=0 Oct 31 23:41:22.894599 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:22.894229 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" event={"ID":"69beb65f-fee0-4fcd-a000-a36b6f45ac4b","Type":"ContainerDied","Data":"017b50046e2f293e1dd0916df4908d9292a3a136d9ff4519f2d4660460d7c415"} Oct 31 23:41:23.006656 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:23.006623 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:23.612331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:23.612304 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:24.049444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.049413 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:24.203365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.203274 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca\") pod \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " Oct 31 23:41:24.203365 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.203333 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca\") pod \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " Oct 31 23:41:24.203616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.203375 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb\" (UniqueName: \"kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb\") pod \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\" (UID: \"69beb65f-fee0-4fcd-a000-a36b6f45ac4b\") " Oct 31 23:41:24.203839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.203751 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca" pod "69beb65f-fee0-4fcd-a000-a36b6f45ac4b" (UID: "69beb65f-fee0-4fcd-a000-a36b6f45ac4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:41:24.205390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.205365 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb" pod "69beb65f-fee0-4fcd-a000-a36b6f45ac4b" (UID: "69beb65f-fee0-4fcd-a000-a36b6f45ac4b"). InnerVolumeSpecName "kube-api-access-2bmbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:41:24.205390 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.205378 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca" pod "69beb65f-fee0-4fcd-a000-a36b6f45ac4b" (UID: "69beb65f-fee0-4fcd-a000-a36b6f45ac4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:41:24.217292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.215828 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:24.304758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.304716 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:24.304758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.304755 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:24.304994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.304773 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmbb\" (UniqueName: \"kubernetes.io/projected/69beb65f-fee0-4fcd-a000-a36b6f45ac4b-kube-api-access-2bmbb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:24.885305 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.885265 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:24.908023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.907994 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" Oct 31 23:41:24.908200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.908066 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj" event={"ID":"69beb65f-fee0-4fcd-a000-a36b6f45ac4b","Type":"ContainerDied","Data":"4c989d577ad2871c3490621211b8b35871681b38b401d4e19c7127e073f4feb4"} Oct 31 23:41:24.908200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:24.908162 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c989d577ad2871c3490621211b8b35871681b38b401d4e19c7127e073f4feb4" Oct 31 23:41:25.506839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:25.506784 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:26.106134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:26.106083 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:26.705812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:26.705757 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:27.307102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:27.307070 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:28.034753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:28.034716 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:28.612040 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:28.612002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:29.207354 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:29.207318 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:29.801048 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:29.801016 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:30.417466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:30.417420 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:31.016525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:31.016478 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:31.623089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:31.623014 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:32.027889 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.027847 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:32.155570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.155527 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls"] Oct 31 23:41:32.156017 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.156002 5043 cpu_manager.go:414] "containerMap: removing container" podUID="69beb65f-fee0-4fcd-a000-a36b6f45ac4b" containerName="revoke-keys" Oct 31 23:41:32.156084 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.156018 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69beb65f-fee0-4fcd-a000-a36b6f45ac4b" containerName="revoke-keys" Oct 31 23:41:32.156147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.156138 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="69beb65f-fee0-4fcd-a000-a36b6f45ac4b" containerName="revoke-keys" Oct 31 23:41:32.159596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.159576 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.164386 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.164362 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:41:32.165400 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.165380 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:41:32.165528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.165380 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:41:32.165660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.165445 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:41:32.169610 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.169586 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls"] Oct 31 23:41:32.285455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.285369 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpjrr\" (UniqueName: \"kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.285455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.285425 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.285455 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.285446 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.388377 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.387921 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpjrr\" (UniqueName: \"kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.388864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.388825 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.389023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.388887 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.389679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.389656 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.392459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.392407 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.401660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.401631 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpjrr\" (UniqueName: \"kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.471066 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.471019 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:32.474829 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.474796 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:32.757555 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.757521 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls"] Oct 31 23:41:32.759822 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:41:32.759791 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56721ec_0135_4667_bdaa_9ceee1d6bf2f.slice/crio-63782f246907be3b8f34c0addb7edd347499508e441f31395c848d3fbe548b50 WatchSource:0}: Error finding container 63782f246907be3b8f34c0addb7edd347499508e441f31395c848d3fbe548b50: Status 404 returned error can't find the container with id 63782f246907be3b8f34c0addb7edd347499508e441f31395c848d3fbe548b50 Oct 31 23:41:32.953561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:32.953519 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" event={"ID":"f56721ec-0135-4667-bdaa-9ceee1d6bf2f","Type":"ContainerStarted","Data":"63782f246907be3b8f34c0addb7edd347499508e441f31395c848d3fbe548b50"} Oct 31 23:41:33.190641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:33.190601 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:33.812678 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:33.812485 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:33.961481 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:33.961450 5043 generic.go:356] "Generic (PLEG): container finished" podID="f56721ec-0135-4667-bdaa-9ceee1d6bf2f" containerID="d619f65ff15370c4e0683f6f26ba18f8aac10be471f41e7b7fe92718235cc1ec" exitCode=0 Oct 31 23:41:33.961671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:33.961513 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" event={"ID":"f56721ec-0135-4667-bdaa-9ceee1d6bf2f","Type":"ContainerDied","Data":"d619f65ff15370c4e0683f6f26ba18f8aac10be471f41e7b7fe92718235cc1ec"} Oct 31 23:41:34.414560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:34.414521 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:35.022321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.022238 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:35.134426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.134398 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:35.315819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.315782 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr\" (UniqueName: \"kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr\") pod \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " Oct 31 23:41:35.315993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.315856 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca\") pod \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " Oct 31 23:41:35.315993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.315956 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca\") pod \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\" (UID: \"f56721ec-0135-4667-bdaa-9ceee1d6bf2f\") " Oct 31 23:41:35.316338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.316307 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca" pod "f56721ec-0135-4667-bdaa-9ceee1d6bf2f" (UID: "f56721ec-0135-4667-bdaa-9ceee1d6bf2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:41:35.317978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.317949 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr" pod "f56721ec-0135-4667-bdaa-9ceee1d6bf2f" (UID: "f56721ec-0135-4667-bdaa-9ceee1d6bf2f"). InnerVolumeSpecName "kube-api-access-hpjrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:41:35.318297 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.318280 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca" pod "f56721ec-0135-4667-bdaa-9ceee1d6bf2f" (UID: "f56721ec-0135-4667-bdaa-9ceee1d6bf2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:41:35.417671 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.417635 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:35.417834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.417696 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:35.417834 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.417714 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpjrr\" (UniqueName: \"kubernetes.io/projected/f56721ec-0135-4667-bdaa-9ceee1d6bf2f-kube-api-access-hpjrr\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:35.647619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.647505 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:35.976810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.976779 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" Oct 31 23:41:35.976810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.976795 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls" event={"ID":"f56721ec-0135-4667-bdaa-9ceee1d6bf2f","Type":"ContainerDied","Data":"63782f246907be3b8f34c0addb7edd347499508e441f31395c848d3fbe548b50"} Oct 31 23:41:35.977001 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:35.976843 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63782f246907be3b8f34c0addb7edd347499508e441f31395c848d3fbe548b50" Oct 31 23:41:36.221157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:36.221060 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:36.813851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:36.813812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:37.116962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:37.116872 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:37.409756 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:37.409672 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:38.001330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:38.001291 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:38.716843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:38.716808 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:39.305196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:39.305161 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:39.917520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:39.917487 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:40.454536 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:41:40.454500 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:41:40.500554 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:40.500503 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:41.115582 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:41.115519 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:41.706945 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:41.706908 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:42.280038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.280004 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f"] Oct 31 23:41:42.296604 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.296571 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6"] Oct 31 23:41:42.336451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.336417 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:42.473046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.473008 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:42.564682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.564585 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls"] Oct 31 23:41:42.567868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.567838 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abd9b5-72kls"] Oct 31 23:41:42.610884 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.610857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:42.741200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:42.741163 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:43.163851 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:43.163812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:43.936740 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:43.936700 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56721ec-0135-4667-bdaa-9ceee1d6bf2f" path="/var/lib/kubelet/pods/f56721ec-0135-4667-bdaa-9ceee1d6bf2f/volumes" Oct 31 23:41:47.281246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:47.281202 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerName="payload-processing" containerID="cri-o://29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701" gracePeriod=25 Oct 31 23:41:47.297784 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:47.297748 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerName="payload-pre-processing" containerID="cri-o://20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18" gracePeriod=25 Oct 31 23:41:47.592067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:47.591988 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:41:47.592242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:47.591986 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:41:49.008732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.008698 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:49.145173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.145139 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:49.308489 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.308394 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4"] Oct 31 23:41:49.308729 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.308697 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" containerID="cri-o://9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d" gracePeriod=30 Oct 31 23:41:49.389078 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.389046 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:49.781788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.781756 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:49.925135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:49.925102 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:53.231526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:53.231484 5043 patch_prober.go:29] interesting pod/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:41:53.231922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:53.231567 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:41:53.976192 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:53.976150 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:41:54.572451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.572389 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:41:54.619011 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.618985 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619061 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619170 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619095 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619234 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619301 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619290 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619421 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619368 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:41:54.619728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619574 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:41:54.619728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619582 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619662 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.619978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619944 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:41:54.619978 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.619965 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:41:54.620099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620066 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.620168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620122 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs\") pod \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\" (UID: \"3a68ce20-595c-436d-8cd1-7a4d2505f01d\") " Oct 31 23:41:54.620621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620508 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:41:54.620621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620523 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.620621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620572 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.620621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620589 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.620621 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.620604 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.621708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.621673 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:41:54.621836 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.621785 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:41:54.622023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.621999 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "kube-api-access-n9h8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:41:54.622637 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.622618 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo" pod "3a68ce20-595c-436d-8cd1-7a4d2505f01d" (UID: "3a68ce20-595c-436d-8cd1-7a4d2505f01d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:41:54.721936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.721907 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.721936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.721932 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.721936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.721941 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.722194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.721951 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a68ce20-595c-436d-8cd1-7a4d2505f01d-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:54.722194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:54.721961 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9h8w\" (UniqueName: \"kubernetes.io/projected/3a68ce20-595c-436d-8cd1-7a4d2505f01d-kube-api-access-n9h8w\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:41:55.109631 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.109592 5043 generic.go:356] "Generic (PLEG): container finished" podID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerID="9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d" exitCode=0 Oct 31 23:41:55.109814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.109642 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" event={"ID":"3a68ce20-595c-436d-8cd1-7a4d2505f01d","Type":"ContainerDied","Data":"9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d"} Oct 31 23:41:55.109814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.109685 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" event={"ID":"3a68ce20-595c-436d-8cd1-7a4d2505f01d","Type":"ContainerDied","Data":"83ff1483dd4e58c3d0062ab671cea2608a172e1ad7fc34b4542a4982536dc067"} Oct 31 23:41:55.109814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.109705 5043 scope.go:122] "RemoveContainer" containerID="9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d" Oct 31 23:41:55.109814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.109648 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4" Oct 31 23:41:55.120457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.120437 5043 scope.go:122] "RemoveContainer" containerID="6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e" Oct 31 23:41:55.130306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.130283 5043 scope.go:122] "RemoveContainer" containerID="9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d" Oct 31 23:41:55.130638 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:41:55.130612 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d\": container with ID starting with 9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d not found: ID does not exist" containerID="9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d" Oct 31 23:41:55.130754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.130644 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d"} err="failed to get container status \"9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d\": rpc error: code = NotFound desc = could not find container \"9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d\": container with ID starting with 9cb5625e2b8bc6c9b77fd045fb196ed1d39dc27ddae681fbde5c430ea5fd278d not found: ID does not exist" Oct 31 23:41:55.130754 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.130662 5043 scope.go:122] "RemoveContainer" containerID="6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e" Oct 31 23:41:55.130952 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:41:55.130923 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e\": container with ID starting with 6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e not found: ID does not exist" containerID="6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e" Oct 31 23:41:55.131012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.130963 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e"} err="failed to get container status \"6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e\": rpc error: code = NotFound desc = could not find container \"6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e\": container with ID starting with 6f746a281206f7998edac37d71749fa1a38d59e8466be8a1ae99b72d2c447f0e not found: ID does not exist" Oct 31 23:41:55.138188 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.138163 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4"] Oct 31 23:41:55.141909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.141886 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-abd9b5-openshift-default-587ff99b89zznr4"] Oct 31 23:41:55.936394 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:55.936365 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" path="/var/lib/kubelet/pods/3a68ce20-595c-436d-8cd1-7a4d2505f01d/volumes" Oct 31 23:41:57.610595 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:57.610528 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:41:57.611727 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:41:57.611675 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 23:42:07.551559 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.551493 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:42:07.574883 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.574859 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:42:07.696373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.696341 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5\" (UniqueName: \"kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5\") pod \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " Oct 31 23:42:07.696539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.696397 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume\") pod \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\" (UID: \"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1\") " Oct 31 23:42:07.696539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.696421 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp\" (UniqueName: \"kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp\") pod \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " Oct 31 23:42:07.696539 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.696450 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume\") pod \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\" (UID: \"0ad3fc5d-f250-48e9-864d-b84466de0a6d\") " Oct 31 23:42:07.696880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.696856 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume" pod "aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" (UID: "aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:42:07.696944 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.696877 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume" pod "0ad3fc5d-f250-48e9-864d-b84466de0a6d" (UID: "0ad3fc5d-f250-48e9-864d-b84466de0a6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:42:07.698495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.698477 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5" pod "aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" (UID: "aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1"). InnerVolumeSpecName "kube-api-access-vbbl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:42:07.698562 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.698513 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp" pod "0ad3fc5d-f250-48e9-864d-b84466de0a6d" (UID: "0ad3fc5d-f250-48e9-864d-b84466de0a6d"). InnerVolumeSpecName "kube-api-access-8hnvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:42:07.798215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.798191 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbl5\" (UniqueName: \"kubernetes.io/projected/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-kube-api-access-vbbl5\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:07.798215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.798214 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:07.798353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.798223 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hnvp\" (UniqueName: \"kubernetes.io/projected/0ad3fc5d-f250-48e9-864d-b84466de0a6d-kube-api-access-8hnvp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:07.798353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:07.798231 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad3fc5d-f250-48e9-864d-b84466de0a6d-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:08.252649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.252613 5043 generic.go:356] "Generic (PLEG): container finished" podID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerID="29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701" exitCode=0 Oct 31 23:42:08.252825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.252665 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" Oct 31 23:42:08.252825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.252672 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" event={"ID":"0ad3fc5d-f250-48e9-864d-b84466de0a6d","Type":"ContainerDied","Data":"29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701"} Oct 31 23:42:08.252825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.252728 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f" event={"ID":"0ad3fc5d-f250-48e9-864d-b84466de0a6d","Type":"ContainerDied","Data":"6f9befe63a1b7e6849eece9296f42533a1b765c23a9466182a5f94b1faa01a68"} Oct 31 23:42:08.252825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.252749 5043 scope.go:122] "RemoveContainer" containerID="29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701" Oct 31 23:42:08.254116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.254094 5043 generic.go:356] "Generic (PLEG): container finished" podID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerID="20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18" exitCode=0 Oct 31 23:42:08.254248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.254123 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" event={"ID":"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1","Type":"ContainerDied","Data":"20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18"} Oct 31 23:42:08.254248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.254151 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" Oct 31 23:42:08.254248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.254161 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6" event={"ID":"aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1","Type":"ContainerDied","Data":"e78d77c933f44dbc67a0a6949557a7fbd9ef98b874cef37161e94947f783df06"} Oct 31 23:42:08.262241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.262224 5043 scope.go:122] "RemoveContainer" containerID="29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701" Oct 31 23:42:08.262516 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:42:08.262495 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701\": container with ID starting with 29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701 not found: ID does not exist" containerID="29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701" Oct 31 23:42:08.262596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.262526 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701"} err="failed to get container status \"29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701\": rpc error: code = NotFound desc = could not find container \"29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701\": container with ID starting with 29ed7d7fa726e36b01a242bb84ce81643dda31e34c01ee9245b263fd0250c701 not found: ID does not exist" Oct 31 23:42:08.262596 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.262558 5043 scope.go:122] "RemoveContainer" containerID="20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18" Oct 31 23:42:08.270966 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.270940 5043 scope.go:122] "RemoveContainer" containerID="20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18" Oct 31 23:42:08.271243 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:42:08.271222 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18\": container with ID starting with 20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18 not found: ID does not exist" containerID="20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18" Oct 31 23:42:08.271298 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.271249 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18"} err="failed to get container status \"20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18\": rpc error: code = NotFound desc = could not find container \"20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18\": container with ID starting with 20c3d84dc1ff18e2e1ad4b67a64f2c9d237a805dbbd44bdd8561f9a624edfa18 not found: ID does not exist" Oct 31 23:42:08.271434 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.271418 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f"] Oct 31 23:42:08.275485 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.275464 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-abd9b5-6c8848f555sl4f"] Oct 31 23:42:08.286500 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.286480 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6"] Oct 31 23:42:08.290600 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:08.290581 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abd9b5-6847cpk2k6"] Oct 31 23:42:09.935111 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:09.935077 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" path="/var/lib/kubelet/pods/0ad3fc5d-f250-48e9-864d-b84466de0a6d/volumes" Oct 31 23:42:09.935505 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:09.935471 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" path="/var/lib/kubelet/pods/aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1/volumes" Oct 31 23:42:10.456273 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:42:10.456241 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:42:10.496852 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:10.496821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:13.288723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:13.288688 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:14.410067 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:14.410030 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:15.770569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:15.769447 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:20.351457 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:20.351370 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:27.428267 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428231 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf"] Oct 31 23:42:27.428874 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428853 5043 cpu_manager.go:414] "containerMap: removing container" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerName="payload-pre-processing" Oct 31 23:42:27.428964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428876 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerName="payload-pre-processing" Oct 31 23:42:27.428964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428897 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" Oct 31 23:42:27.428964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428905 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" Oct 31 23:42:27.428964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428950 5043 cpu_manager.go:414] "containerMap: removing container" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerName="payload-processing" Oct 31 23:42:27.428964 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428958 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerName="payload-processing" Oct 31 23:42:27.429227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428978 5043 cpu_manager.go:414] "containerMap: removing container" podUID="f56721ec-0135-4667-bdaa-9ceee1d6bf2f" containerName="revoke-keys" Oct 31 23:42:27.429227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.428986 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56721ec-0135-4667-bdaa-9ceee1d6bf2f" containerName="revoke-keys" Oct 31 23:42:27.429227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.429099 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad3fc5d-f250-48e9-864d-b84466de0a6d" containerName="payload-processing" Oct 31 23:42:27.429227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.429115 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" Oct 31 23:42:27.429227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.429125 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56721ec-0135-4667-bdaa-9ceee1d6bf2f" containerName="revoke-keys" Oct 31 23:42:27.429227 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.429136 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeba8c72-4fe4-45e5-a5a9-f63d74cdecd1" containerName="payload-pre-processing" Oct 31 23:42:27.433723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.433702 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.439387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.439359 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:42:27.439574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.439435 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:42:27.441257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.440371 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:42:27.441257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.440457 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:42:27.441257 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.441150 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf"] Oct 31 23:42:27.511324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.511294 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjlg\" (UniqueName: \"kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.511324 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.511326 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.511518 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.511498 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.612072 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.612041 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.612246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.612087 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjlg\" (UniqueName: \"kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.612246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.612109 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.612773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.612747 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.614872 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.614847 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.626006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.625981 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjlg\" (UniqueName: \"kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg\") pod \"maas-api-revoke-sub-452b6d7b-76wcf\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:27.755580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:27.755541 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:28.035570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:28.035469 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf"] Oct 31 23:42:28.040713 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:42:28.040686 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada9a6e4_9b55_4945_9a83_9e5984df04d7.slice/crio-43f5fe9ebaabb5f2abcb41cddbbcb99aad2e0a5083ce1bec7a31994d6a21bc11 WatchSource:0}: Error finding container 43f5fe9ebaabb5f2abcb41cddbbcb99aad2e0a5083ce1bec7a31994d6a21bc11: Status 404 returned error can't find the container with id 43f5fe9ebaabb5f2abcb41cddbbcb99aad2e0a5083ce1bec7a31994d6a21bc11 Oct 31 23:42:28.447699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:28.446736 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" event={"ID":"ada9a6e4-9b55-4945-9a83-9e5984df04d7","Type":"ContainerStarted","Data":"7f892a7a9b36b36e8da769cdb7e74b193d9602c41d99e22b86cb2b99c8fe7d8f"} Oct 31 23:42:28.447699 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:28.446804 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" event={"ID":"ada9a6e4-9b55-4945-9a83-9e5984df04d7","Type":"ContainerStarted","Data":"43f5fe9ebaabb5f2abcb41cddbbcb99aad2e0a5083ce1bec7a31994d6a21bc11"} Oct 31 23:42:28.468494 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:28.468429 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" podStartSLOduration=1.468411363 podStartE2EDuration="1.468411363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:42:28.465669723 +0000 UTC m=+3174.695224147" watchObservedRunningTime="2026-10-31 23:42:28.468411363 +0000 UTC m=+3174.697965774" Oct 31 23:42:29.580568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:29.576805 5043 generic.go:356] "Generic (PLEG): container finished" podID="ada9a6e4-9b55-4945-9a83-9e5984df04d7" containerID="7f892a7a9b36b36e8da769cdb7e74b193d9602c41d99e22b86cb2b99c8fe7d8f" exitCode=0 Oct 31 23:42:29.580568 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:29.577037 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" event={"ID":"ada9a6e4-9b55-4945-9a83-9e5984df04d7","Type":"ContainerDied","Data":"7f892a7a9b36b36e8da769cdb7e74b193d9602c41d99e22b86cb2b99c8fe7d8f"} Oct 31 23:42:31.415574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.411752 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:31.503566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.503175 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca\") pod \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " Oct 31 23:42:31.503566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.503274 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca\") pod \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " Oct 31 23:42:31.503566 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.503370 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg\" (UniqueName: \"kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg\") pod \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\" (UID: \"ada9a6e4-9b55-4945-9a83-9e5984df04d7\") " Oct 31 23:42:31.513574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.504685 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca" pod "ada9a6e4-9b55-4945-9a83-9e5984df04d7" (UID: "ada9a6e4-9b55-4945-9a83-9e5984df04d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:42:31.580095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.580031 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca" pod "ada9a6e4-9b55-4945-9a83-9e5984df04d7" (UID: "ada9a6e4-9b55-4945-9a83-9e5984df04d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:42:31.580461 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.580415 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg" pod "ada9a6e4-9b55-4945-9a83-9e5984df04d7" (UID: "ada9a6e4-9b55-4945-9a83-9e5984df04d7"). InnerVolumeSpecName "kube-api-access-hkjlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:42:31.606317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.606226 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkjlg\" (UniqueName: \"kubernetes.io/projected/ada9a6e4-9b55-4945-9a83-9e5984df04d7-kube-api-access-hkjlg\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:31.606317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.606259 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:31.606317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.606276 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada9a6e4-9b55-4945-9a83-9e5984df04d7-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:31.626641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.612781 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" event={"ID":"ada9a6e4-9b55-4945-9a83-9e5984df04d7","Type":"ContainerDied","Data":"43f5fe9ebaabb5f2abcb41cddbbcb99aad2e0a5083ce1bec7a31994d6a21bc11"} Oct 31 23:42:31.626641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.612841 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f5fe9ebaabb5f2abcb41cddbbcb99aad2e0a5083ce1bec7a31994d6a21bc11" Oct 31 23:42:31.626641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:31.612912 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf" Oct 31 23:42:38.398308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:38.398251 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:40.470402 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:42:40.470360 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:42:40.548850 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:40.548805 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:42:41.266768 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:41.266729 5043 scope.go:122] "RemoveContainer" containerID="3d844cc70fd6ec97e6f0759a01aa3d6167bef685546e2342f855298eb0f2651f" Oct 31 23:42:43.731835 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.731799 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw"] Oct 31 23:42:43.733429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.732458 5043 cpu_manager.go:414] "containerMap: removing container" podUID="ada9a6e4-9b55-4945-9a83-9e5984df04d7" containerName="revoke-keys" Oct 31 23:42:43.733429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.732476 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada9a6e4-9b55-4945-9a83-9e5984df04d7" containerName="revoke-keys" Oct 31 23:42:43.733429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.732519 5043 cpu_manager.go:414] "containerMap: removing container" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" Oct 31 23:42:43.733429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.732527 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" Oct 31 23:42:43.733429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.732656 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada9a6e4-9b55-4945-9a83-9e5984df04d7" containerName="revoke-keys" Oct 31 23:42:43.733429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.732669 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a68ce20-595c-436d-8cd1-7a4d2505f01d" containerName="istio-proxy" Oct 31 23:42:43.757325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.740812 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.757325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.746233 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:42:43.757325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.747404 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:42:43.757325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.747536 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:42:43.757325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.747683 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:42:43.757325 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.752106 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw"] Oct 31 23:42:43.865016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.864982 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzfz\" (UniqueName: \"kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.865212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.865104 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.865212 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.865137 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.966263 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.966218 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzfz\" (UniqueName: \"kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.966871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.966842 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.967053 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.967033 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.967928 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.967899 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.979292 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.979246 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzfz\" (UniqueName: \"kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:43.982879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:43.982802 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-768xw\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:44.126116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:44.126076 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:44.438759 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:44.438713 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw"] Oct 31 23:42:44.439885 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:42:44.439845 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e964ec2_2f8c_4295_9698_9b129f2e41c5.slice/crio-ca5d0a3f7e5104db00f3548f3bc0b622303c94b4b63b5b477b30e12947da7593 WatchSource:0}: Error finding container ca5d0a3f7e5104db00f3548f3bc0b622303c94b4b63b5b477b30e12947da7593: Status 404 returned error can't find the container with id ca5d0a3f7e5104db00f3548f3bc0b622303c94b4b63b5b477b30e12947da7593 Oct 31 23:42:44.768570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:44.766192 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" event={"ID":"4e964ec2-2f8c-4295-9698-9b129f2e41c5","Type":"ContainerStarted","Data":"ca5d0a3f7e5104db00f3548f3bc0b622303c94b4b63b5b477b30e12947da7593"} Oct 31 23:42:45.811894 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:45.811801 5043 generic.go:356] "Generic (PLEG): container finished" podID="4e964ec2-2f8c-4295-9698-9b129f2e41c5" containerID="fe399f6fb336e039d71c36712ef6433e1d3f669da76d79420bf9df5b49cdd09f" exitCode=0 Oct 31 23:42:45.812478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:45.812448 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" event={"ID":"4e964ec2-2f8c-4295-9698-9b129f2e41c5","Type":"ContainerDied","Data":"fe399f6fb336e039d71c36712ef6433e1d3f669da76d79420bf9df5b49cdd09f"} Oct 31 23:42:46.960042 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.960012 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:46.994222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.994187 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca\") pod \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " Oct 31 23:42:46.994383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.994303 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca\") pod \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " Oct 31 23:42:46.994383 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.994378 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz\" (UniqueName: \"kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz\") pod \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\" (UID: \"4e964ec2-2f8c-4295-9698-9b129f2e41c5\") " Oct 31 23:42:46.995030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.994628 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca" pod "4e964ec2-2f8c-4295-9698-9b129f2e41c5" (UID: "4e964ec2-2f8c-4295-9698-9b129f2e41c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:42:46.995030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.994762 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:46.996891 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.996864 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca" pod "4e964ec2-2f8c-4295-9698-9b129f2e41c5" (UID: "4e964ec2-2f8c-4295-9698-9b129f2e41c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:42:46.996989 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:46.996875 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz" pod "4e964ec2-2f8c-4295-9698-9b129f2e41c5" (UID: "4e964ec2-2f8c-4295-9698-9b129f2e41c5"). InnerVolumeSpecName "kube-api-access-5mzfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:42:47.095761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:47.095674 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e964ec2-2f8c-4295-9698-9b129f2e41c5-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:47.095761 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:47.095709 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mzfz\" (UniqueName: \"kubernetes.io/projected/4e964ec2-2f8c-4295-9698-9b129f2e41c5-kube-api-access-5mzfz\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:42:47.827642 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:47.827608 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" Oct 31 23:42:47.827816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:47.827616 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw" event={"ID":"4e964ec2-2f8c-4295-9698-9b129f2e41c5","Type":"ContainerDied","Data":"ca5d0a3f7e5104db00f3548f3bc0b622303c94b4b63b5b477b30e12947da7593"} Oct 31 23:42:47.827816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:47.827742 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5d0a3f7e5104db00f3548f3bc0b622303c94b4b63b5b477b30e12947da7593" Oct 31 23:42:53.023517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:53.023476 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c"] Oct 31 23:42:53.026366 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:53.026337 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z9g6c"] Oct 31 23:42:53.934674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:53.934633 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286dd146-aa4a-4d43-9324-eae87b6bfc88" path="/var/lib/kubelet/pods/286dd146-aa4a-4d43-9324-eae87b6bfc88/volumes" Oct 31 23:42:56.873569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:56.872694 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-fw9np container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:42:56.873569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:42:56.872788 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-fw9np" podUID="c08d1d39-5bdc-4834-a0de-af51ef202a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:43:04.274272 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.274232 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7"] Oct 31 23:43:04.274726 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.274703 5043 cpu_manager.go:414] "containerMap: removing container" podUID="4e964ec2-2f8c-4295-9698-9b129f2e41c5" containerName="revoke-keys" Oct 31 23:43:04.274769 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.274727 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e964ec2-2f8c-4295-9698-9b129f2e41c5" containerName="revoke-keys" Oct 31 23:43:04.274814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.274805 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e964ec2-2f8c-4295-9698-9b129f2e41c5" containerName="revoke-keys" Oct 31 23:43:04.279284 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.279263 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.284089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.284061 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 23:43:04.284215 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.284125 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:43:04.285724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.285693 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:43:04.285839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.285728 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-67hcr\"" Oct 31 23:43:04.286329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.286308 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7"] Oct 31 23:43:04.380752 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.380718 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.380933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.380792 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.380933 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.380872 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppwb\" (UniqueName: \"kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.481903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.481874 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.482037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.481919 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.482037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.481947 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppwb\" (UniqueName: \"kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.482580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.482559 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.484197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.484175 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.494807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.494783 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppwb\" (UniqueName: \"kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb\") pod \"maas-api-revoke-sub-338bf790-mqjd7\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.594507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.594455 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:04.874140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:04.874110 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7"] Oct 31 23:43:04.876068 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:43:04.876041 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a08599_6bca_4f6a_be22_d7ea314ed744.slice/crio-32bdef5eddd9b20530b1a2bf4958290b2f620c2f06158de049ed0baef01f16bf WatchSource:0}: Error finding container 32bdef5eddd9b20530b1a2bf4958290b2f620c2f06158de049ed0baef01f16bf: Status 404 returned error can't find the container with id 32bdef5eddd9b20530b1a2bf4958290b2f620c2f06158de049ed0baef01f16bf Oct 31 23:43:05.082268 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:05.082231 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" event={"ID":"c6a08599-6bca-4f6a-be22-d7ea314ed744","Type":"ContainerStarted","Data":"83f7e83540808a41babdd236c56405fc24238628ae0054d2235868358bde9254"} Oct 31 23:43:05.082422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:05.082294 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" event={"ID":"c6a08599-6bca-4f6a-be22-d7ea314ed744","Type":"ContainerStarted","Data":"32bdef5eddd9b20530b1a2bf4958290b2f620c2f06158de049ed0baef01f16bf"} Oct 31 23:43:05.102882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:05.102840 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" podStartSLOduration=1.102827101 podStartE2EDuration="1.102827101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:05.100940787 +0000 UTC m=+3211.330495207" watchObservedRunningTime="2026-10-31 23:43:05.102827101 +0000 UTC m=+3211.332381511" Oct 31 23:43:06.088881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:06.088844 5043 generic.go:356] "Generic (PLEG): container finished" podID="c6a08599-6bca-4f6a-be22-d7ea314ed744" containerID="83f7e83540808a41babdd236c56405fc24238628ae0054d2235868358bde9254" exitCode=0 Oct 31 23:43:06.089270 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:06.088918 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" event={"ID":"c6a08599-6bca-4f6a-be22-d7ea314ed744","Type":"ContainerDied","Data":"83f7e83540808a41babdd236c56405fc24238628ae0054d2235868358bde9254"} Oct 31 23:43:07.261936 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.261907 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:07.409827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.409738 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca\") pod \"c6a08599-6bca-4f6a-be22-d7ea314ed744\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " Oct 31 23:43:07.410009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.409892 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb\" (UniqueName: \"kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb\") pod \"c6a08599-6bca-4f6a-be22-d7ea314ed744\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " Oct 31 23:43:07.410009 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.409916 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca\") pod \"c6a08599-6bca-4f6a-be22-d7ea314ed744\" (UID: \"c6a08599-6bca-4f6a-be22-d7ea314ed744\") " Oct 31 23:43:07.410312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.410280 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca" pod "c6a08599-6bca-4f6a-be22-d7ea314ed744" (UID: "c6a08599-6bca-4f6a-be22-d7ea314ed744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:43:07.411843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.411813 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca" pod "c6a08599-6bca-4f6a-be22-d7ea314ed744" (UID: "c6a08599-6bca-4f6a-be22-d7ea314ed744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:43:07.411971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.411954 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb" pod "c6a08599-6bca-4f6a-be22-d7ea314ed744" (UID: "c6a08599-6bca-4f6a-be22-d7ea314ed744"). InnerVolumeSpecName "kube-api-access-sppwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:43:07.510616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.510583 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppwb\" (UniqueName: \"kubernetes.io/projected/c6a08599-6bca-4f6a-be22-d7ea314ed744-kube-api-access-sppwb\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:07.510616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.510611 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-openshift-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:07.510616 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:07.510622 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a08599-6bca-4f6a-be22-d7ea314ed744-maas-api-xks-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:08.109426 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:08.109395 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" event={"ID":"c6a08599-6bca-4f6a-be22-d7ea314ed744","Type":"ContainerDied","Data":"32bdef5eddd9b20530b1a2bf4958290b2f620c2f06158de049ed0baef01f16bf"} Oct 31 23:43:08.109617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:08.109441 5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32bdef5eddd9b20530b1a2bf4958290b2f620c2f06158de049ed0baef01f16bf" Oct 31 23:43:08.109617 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:08.109404 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7" Oct 31 23:43:10.455468 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:43:10.455432 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:43:16.499197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.499166 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr"] Oct 31 23:43:16.499713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.499625 5043 cpu_manager.go:414] "containerMap: removing container" podUID="c6a08599-6bca-4f6a-be22-d7ea314ed744" containerName="revoke-keys" Oct 31 23:43:16.499713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.499638 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a08599-6bca-4f6a-be22-d7ea314ed744" containerName="revoke-keys" Oct 31 23:43:16.499806 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.499735 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a08599-6bca-4f6a-be22-d7ea314ed744" containerName="revoke-keys" Oct 31 23:43:16.503135 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.503117 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.513353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.513292 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 23:43:16.513353 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.513316 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 23:43:16.515432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.515411 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 23:43:16.515606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.515442 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7j72\"" Oct 31 23:43:16.515682 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.515485 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 23:43:16.518876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.518254 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr"] Oct 31 23:43:16.602911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.602884 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.603114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.602920 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.603114 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.603075 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldbs\" (UniqueName: \"kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.603216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.603116 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.703814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.703781 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldbs\" (UniqueName: \"kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.703814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.703818 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.704056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.703872 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.704056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.703892 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.704623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.704600 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.706263 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.706238 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.706376 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.706357 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.716149 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.716126 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldbs\" (UniqueName: \"kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs\") pod \"maas-api-6784f6bdbc-q2sdr\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:16.820911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:16.820847 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:17.101903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:17.101876 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr"] Oct 31 23:43:17.104743 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:43:17.104716 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1637113d_c047_4e66_8de8_a5317c6916b9.slice/crio-2e8f4444e669a29cda1d6acc83c2ba9feb3c6f87043bc3cbce79f0eb7f579ce8 WatchSource:0}: Error finding container 2e8f4444e669a29cda1d6acc83c2ba9feb3c6f87043bc3cbce79f0eb7f579ce8: Status 404 returned error can't find the container with id 2e8f4444e669a29cda1d6acc83c2ba9feb3c6f87043bc3cbce79f0eb7f579ce8 Oct 31 23:43:17.106134 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:17.106113 5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 23:43:17.158624 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:17.158586 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" event={"ID":"1637113d-c047-4e66-8de8-a5317c6916b9","Type":"ContainerStarted","Data":"2e8f4444e669a29cda1d6acc83c2ba9feb3c6f87043bc3cbce79f0eb7f579ce8"} Oct 31 23:43:17.629948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:17.629908 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:18.166194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:18.166157 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" event={"ID":"1637113d-c047-4e66-8de8-a5317c6916b9","Type":"ContainerStarted","Data":"2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331"} Oct 31 23:43:18.166385 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:18.166228 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:18.187507 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:18.187457 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" podStartSLOduration=1.704149951 podStartE2EDuration="2.187444555s" totalImagesPullingTime="483.294604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:18.185722934 +0000 UTC m=+3224.415277341" watchObservedRunningTime="2026-10-31 23:43:18.187444555 +0000 UTC m=+3224.416998964" Oct 31 23:43:23.583065 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:23.583003 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:23.941839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:23.941753 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb"] Oct 31 23:43:23.950814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:23.950779 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb"] Oct 31 23:43:23.950979 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:23.950917 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:23.955570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:23.955531 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3d08af8f-openshift-default-dockercfg-pxtgx\"" Oct 31 23:43:24.017030 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.017000 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:24.072116 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072089 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072126 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072148 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072289 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072331 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072320 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072351 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072460 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072367 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkvp\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072472 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.072525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.072509 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.154520 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.154477 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:24.173800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173772 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.173984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173806 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.173984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173830 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.173984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173886 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.173984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173923 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.173984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173962 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.174280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.173986 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkvp\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.174280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.174044 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.174280 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.174082 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.174431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.174277 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.174786 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.174761 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.174969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.174787 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.175117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.175006 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.175316 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.175294 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.176760 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.176737 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.177124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.177104 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.178016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.177997 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:24.186102 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.186062 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.186432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.186409 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkvp\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp\") pod \"e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.267763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.267731 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:24.379674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.379640 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf"] Oct 31 23:43:24.384474 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.384445 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.394963 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.394935 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf"] Oct 31 23:43:24.477185 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.477147 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-openshift-service-ca\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.477424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.477211 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-maas-api-tls\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.477424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.477303 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-metrics-certs\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.477424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.477394 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57b46\" (UniqueName: \"kubernetes.io/projected/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-kube-api-access-57b46\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.561269 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.561240 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb"] Oct 31 23:43:24.563335 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:43:24.563302 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86667268_0fab_4537_b06e_16563d324da0.slice/crio-72a0c95452257bb13312d0e9c13c84a52218d1961dae66893e7937da15005bc5 WatchSource:0}: Error finding container 72a0c95452257bb13312d0e9c13c84a52218d1961dae66893e7937da15005bc5: Status 404 returned error can't find the container with id 72a0c95452257bb13312d0e9c13c84a52218d1961dae66893e7937da15005bc5 Oct 31 23:43:24.565535 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.565501 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:43:24.565647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.565578 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:43:24.565647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.565608 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:43:24.578327 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.578303 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57b46\" (UniqueName: \"kubernetes.io/projected/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-kube-api-access-57b46\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.578407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.578385 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-openshift-service-ca\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.578528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.578452 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-maas-api-tls\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.578528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.578521 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-metrics-certs\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.579098 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.579072 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-openshift-service-ca\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.580970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.580948 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-maas-api-tls\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.581079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.580974 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-metrics-certs\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.590531 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.590511 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57b46\" (UniqueName: \"kubernetes.io/projected/8e2ae269-d7ca-4d38-8d6a-d348a4f450be-kube-api-access-57b46\") pod \"maas-api-6bf6d56f9f-hqjpf\" (UID: \"8e2ae269-d7ca-4d38-8d6a-d348a4f450be\") " pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:24.703812 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:24.703775 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:25.000088 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.000058 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf"] Oct 31 23:43:25.005571 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:43:25.004828 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2ae269_d7ca_4d38_8d6a_d348a4f450be.slice/crio-7dfee7eeb34b53684b00b289899dcc67a8efb891bbbbed878b700f26b9a4dc7d WatchSource:0}: Error finding container 7dfee7eeb34b53684b00b289899dcc67a8efb891bbbbed878b700f26b9a4dc7d: Status 404 returned error can't find the container with id 7dfee7eeb34b53684b00b289899dcc67a8efb891bbbbed878b700f26b9a4dc7d Oct 31 23:43:25.205929 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.205894 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" event={"ID":"8e2ae269-d7ca-4d38-8d6a-d348a4f450be","Type":"ContainerStarted","Data":"7dfee7eeb34b53684b00b289899dcc67a8efb891bbbbed878b700f26b9a4dc7d"} Oct 31 23:43:25.207530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.207497 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" event={"ID":"86667268-0fab-4537-b06e-16563d324da0","Type":"ContainerStarted","Data":"244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230"} Oct 31 23:43:25.207670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.207542 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" event={"ID":"86667268-0fab-4537-b06e-16563d324da0","Type":"ContainerStarted","Data":"72a0c95452257bb13312d0e9c13c84a52218d1961dae66893e7937da15005bc5"} Oct 31 23:43:25.232825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.232781 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podStartSLOduration=2.23276911 podStartE2EDuration="2.23276911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:25.229940716 +0000 UTC m=+3231.459495127" watchObservedRunningTime="2026-10-31 23:43:25.23276911 +0000 UTC m=+3231.462323519" Oct 31 23:43:25.268338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.268275 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:25.273467 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:25.273437 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:26.214558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:26.214500 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" event={"ID":"8e2ae269-d7ca-4d38-8d6a-d348a4f450be","Type":"ContainerStarted","Data":"818eb5e1062026ced695b42b13660b8aa6c90e6a6229687dfc1d57d5c8820aff"} Oct 31 23:43:26.215055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:26.214593 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:26.215055 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:26.214988 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:26.216064 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:26.216040 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:26.235159 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:26.235119 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" podStartSLOduration=1.8430252980000001 podStartE2EDuration="2.235109334s" totalImagesPullingTime="392.084036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:43:26.234402108 +0000 UTC m=+3232.463956519" watchObservedRunningTime="2026-10-31 23:43:26.235109334 +0000 UTC m=+3232.464663742" Oct 31 23:43:26.902873 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:26.902839 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:27.051730 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:27.051682 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:27.997586 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:27.997536 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:28.149337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:28.149301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:28.369483 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:28.369360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:28.804035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:28.803993 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:29.462296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:29.462259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:30.128046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:30.128006 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:30.820332 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:30.820292 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:31.428569 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:31.424707 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:32.041178 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.041133 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:32.243791 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.243762 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6bf6d56f9f-hqjpf" Oct 31 23:43:32.298838 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.298748 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr"] Oct 31 23:43:32.299034 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.299001 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" podUID="1637113d-c047-4e66-8de8-a5317c6916b9" containerName="maas-api" containerID="cri-o://2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331" gracePeriod=30 Oct 31 23:43:32.601016 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.600994 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:32.607636 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.607595 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:32.678249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.678198 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs\" (UniqueName: \"kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs\") pod \"1637113d-c047-4e66-8de8-a5317c6916b9\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " Oct 31 23:43:32.678425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.678273 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca\") pod \"1637113d-c047-4e66-8de8-a5317c6916b9\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " Oct 31 23:43:32.678425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.678350 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs\") pod \"1637113d-c047-4e66-8de8-a5317c6916b9\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " Oct 31 23:43:32.678425 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.678417 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls\") pod \"1637113d-c047-4e66-8de8-a5317c6916b9\" (UID: \"1637113d-c047-4e66-8de8-a5317c6916b9\") " Oct 31 23:43:32.679503 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.679472 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca" pod "1637113d-c047-4e66-8de8-a5317c6916b9" (UID: "1637113d-c047-4e66-8de8-a5317c6916b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:43:32.681743 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.681711 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls" pod "1637113d-c047-4e66-8de8-a5317c6916b9" (UID: "1637113d-c047-4e66-8de8-a5317c6916b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:43:32.682340 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.682305 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs" pod "1637113d-c047-4e66-8de8-a5317c6916b9" (UID: "1637113d-c047-4e66-8de8-a5317c6916b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:43:32.682909 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.682881 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs" pod "1637113d-c047-4e66-8de8-a5317c6916b9" (UID: "1637113d-c047-4e66-8de8-a5317c6916b9"). InnerVolumeSpecName "kube-api-access-6ldbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:43:32.779328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.779296 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:32.779328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.779327 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ldbs\" (UniqueName: \"kubernetes.io/projected/1637113d-c047-4e66-8de8-a5317c6916b9-kube-api-access-6ldbs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:32.779509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.779337 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1637113d-c047-4e66-8de8-a5317c6916b9-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:32.779509 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:32.779347 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1637113d-c047-4e66-8de8-a5317c6916b9-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:43:33.255602 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.255539 5043 generic.go:356] "Generic (PLEG): container finished" podID="1637113d-c047-4e66-8de8-a5317c6916b9" containerID="2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331" exitCode=0 Oct 31 23:43:33.256104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.255614 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" Oct 31 23:43:33.256104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.255572 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" event={"ID":"1637113d-c047-4e66-8de8-a5317c6916b9","Type":"ContainerDied","Data":"2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331"} Oct 31 23:43:33.256104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.255720 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr" event={"ID":"1637113d-c047-4e66-8de8-a5317c6916b9","Type":"ContainerDied","Data":"2e8f4444e669a29cda1d6acc83c2ba9feb3c6f87043bc3cbce79f0eb7f579ce8"} Oct 31 23:43:33.256104 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.255757 5043 scope.go:122] "RemoveContainer" containerID="2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331" Oct 31 23:43:33.270484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.270466 5043 scope.go:122] "RemoveContainer" containerID="2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331" Oct 31 23:43:33.270778 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:43:33.270758 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331\": container with ID starting with 2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331 not found: ID does not exist" containerID="2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331" Oct 31 23:43:33.270864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.270787 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331"} err="failed to get container status \"2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331\": rpc error: code = NotFound desc = could not find container \"2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331\": container with ID starting with 2af9f7e7111f6da20d8e8bdeb7b5d83af0d5aead0c81f8684aa70b6cddc3d331 not found: ID does not exist" Oct 31 23:43:33.284337 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.284314 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr"] Oct 31 23:43:33.288168 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.288148 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6784f6bdbc-q2sdr"] Oct 31 23:43:33.335571 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.333433 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:33.947694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:33.947653 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1637113d-c047-4e66-8de8-a5317c6916b9" path="/var/lib/kubelet/pods/1637113d-c047-4e66-8de8-a5317c6916b9/volumes" Oct 31 23:43:34.003304 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:34.003273 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:34.839061 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:34.839031 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:35.493902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:35.493866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:36.011184 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:36.011139 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:36.609736 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:36.609700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:37.303484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:37.303450 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:37.966558 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:37.966520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:38.698191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:38.698158 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:39.315099 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:39.315056 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:40.002432 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.002358 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:40.299058 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.298975 5043 generic.go:356] "Generic (PLEG): container finished" podID="86667268-0fab-4537-b06e-16563d324da0" containerID="244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230" exitCode=0 Oct 31 23:43:40.299447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.299048 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" event={"ID":"86667268-0fab-4537-b06e-16563d324da0","Type":"ContainerDied","Data":"244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230"} Oct 31 23:43:40.299447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.299432 5043 scope.go:122] "RemoveContainer" containerID="244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230" Oct 31 23:43:40.300674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.300639 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:43:40.300798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.300724 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:43:40.300798 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.300765 5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 23:43:40.460255 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:43:40.460220 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:43:40.628271 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:40.628152 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:41.218690 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:41.218654 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:41.300579 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:41.300400 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:41.310708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:41.310670 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" event={"ID":"86667268-0fab-4537-b06e-16563d324da0","Type":"ContainerStarted","Data":"deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef"} Oct 31 23:43:41.910844 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:41.910799 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:42.268609 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:42.268577 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:42.269877 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:42.269848 5043 patch_prober.go:29] interesting pod/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 31 23:43:42.269992 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:42.269909 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 31 23:43:42.508405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:42.508349 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:43.269174 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:43.269136 5043 patch_prober.go:29] interesting pod/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 31 23:43:43.269601 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:43.269202 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 31 23:43:43.304501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:43.304460 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:43.925126 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:43.925077 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:44.268147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:44.268113 5043 patch_prober.go:29] interesting pod/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 31 23:43:44.268328 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:44.268174 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 31 23:43:44.516924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:44.516857 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:45.133575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:45.132653 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:45.268249 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:45.268214 5043 patch_prober.go:29] interesting pod/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 31 23:43:45.268423 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:45.268267 5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 31 23:43:45.741573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:45.735676 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:46.272210 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:46.272180 5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:46.272674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:46.272654 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:46.273330 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:46.273314 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:43:46.314137 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:46.314109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:46.922789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:46.922754 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:47.510758 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:47.510726 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:48.203780 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:48.203741 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:48.700952 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:48.700913 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:49.303912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:49.303721 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:49.912387 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:49.912293 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:50.501921 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:50.501881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:51.109560 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:51.109524 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:51.706355 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:51.706311 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:52.287459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:52.287425 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:52.470163 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:52.470130 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:52.907967 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:52.907937 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:53.049778 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:53.049740 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:53.194343 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:53.194257 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:53.660362 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:53.660317 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:53.799957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:53.799904 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:43:59.125892 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:59.125857 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb"] Oct 31 23:43:59.126475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:59.126058 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" containerID="cri-o://deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef" gracePeriod=30 Oct 31 23:43:59.204424 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:43:59.204366 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:00.563140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:00.563089 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:00.960447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:00.960323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:01.221140 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:01.221007 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:01.274312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:01.274274 5043 patch_prober.go:29] interesting pod/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 23:44:01.274490 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:01.274332 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 23:44:04.382458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.382433 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:44:04.423735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423671 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.423735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423731 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.423922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423772 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.423922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423789 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.423922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423853 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.423922 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423892 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.424157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423927 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.424157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423955 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.424157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.423988 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert\") pod \"86667268-0fab-4537-b06e-16563d324da0\" (UID: \"86667268-0fab-4537-b06e-16563d324da0\") " Oct 31 23:44:04.424157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424081 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:44:04.424157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424142 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:44:04.424431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424208 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:44:04.424431 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424393 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:44:04.424561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424446 5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.424561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424462 5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-data\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.424561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424482 5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-workload-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.424911 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.424887 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:44:04.426358 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.426328 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "kube-api-access-6wkvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:44:04.426464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.426356 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 23:44:04.426464 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.426386 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:44:04.426605 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.426582 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy" pod "86667268-0fab-4537-b06e-16563d324da0" (UID: "86667268-0fab-4537-b06e-16563d324da0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 23:44:04.479915 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.479888 5043 generic.go:356] "Generic (PLEG): container finished" podID="86667268-0fab-4537-b06e-16563d324da0" containerID="deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef" exitCode=0 Oct 31 23:44:04.480007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.479944 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" Oct 31 23:44:04.480007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.479971 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" event={"ID":"86667268-0fab-4537-b06e-16563d324da0","Type":"ContainerDied","Data":"deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef"} Oct 31 23:44:04.480083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.480033 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb" event={"ID":"86667268-0fab-4537-b06e-16563d324da0","Type":"ContainerDied","Data":"72a0c95452257bb13312d0e9c13c84a52218d1961dae66893e7937da15005bc5"} Oct 31 23:44:04.480083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.480062 5043 scope.go:122] "RemoveContainer" containerID="deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef" Oct 31 23:44:04.490022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.490005 5043 scope.go:122] "RemoveContainer" containerID="244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230" Oct 31 23:44:04.498079 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.498061 5043 scope.go:122] "RemoveContainer" containerID="deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef" Oct 31 23:44:04.498349 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:44:04.498332 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef\": container with ID starting with deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef not found: ID does not exist" containerID="deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef" Oct 31 23:44:04.498401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.498357 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef"} err="failed to get container status \"deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef\": rpc error: code = NotFound desc = could not find container \"deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef\": container with ID starting with deaec19afb91ed8b1016a79585a4e7c826bbf31edf23815cb351528bb792abef not found: ID does not exist" Oct 31 23:44:04.498401 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.498375 5043 scope.go:122] "RemoveContainer" containerID="244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230" Oct 31 23:44:04.498632 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:44:04.498609 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230\": container with ID starting with 244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230 not found: ID does not exist" containerID="244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230" Oct 31 23:44:04.498687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.498643 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230"} err="failed to get container status \"244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230\": rpc error: code = NotFound desc = could not find container \"244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230\": container with ID starting with 244e89cb68e6868465e829aac8b07474f7046e25d3edc9bd8b635bb8c604e230 not found: ID does not exist" Oct 31 23:44:04.505530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.505507 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb"] Oct 31 23:44:04.509012 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.508994 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3d08af8f-openshift-default-85586d855c-m54gb"] Oct 31 23:44:04.525444 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.525425 5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-istio-token\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.525525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.525444 5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-istio-envoy\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.525525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.525455 5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86667268-0fab-4537-b06e-16563d324da0-istio-podinfo\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.525525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.525465 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wkvp\" (UniqueName: \"kubernetes.io/projected/86667268-0fab-4537-b06e-16563d324da0-kube-api-access-6wkvp\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.525525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.525474 5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86667268-0fab-4537-b06e-16563d324da0-istiod-ca-cert\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.525525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.525482 5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86667268-0fab-4537-b06e-16563d324da0-credential-socket\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:44:04.753415 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.753389 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:04.888307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:04.888282 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:05.829945 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.829906 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt"] Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830337 5043 cpu_manager.go:414] "containerMap: removing container" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830348 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830358 5043 cpu_manager.go:414] "containerMap: removing container" podUID="1637113d-c047-4e66-8de8-a5317c6916b9" containerName="maas-api" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830363 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1637113d-c047-4e66-8de8-a5317c6916b9" containerName="maas-api" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830370 5043 cpu_manager.go:414] "containerMap: removing container" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830375 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830466 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830478 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1637113d-c047-4e66-8de8-a5317c6916b9" containerName="maas-api" Oct 31 23:44:05.830497 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.830488 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="86667268-0fab-4537-b06e-16563d324da0" containerName="istio-proxy" Oct 31 23:44:05.833810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.833791 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:05.852373 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.852342 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt"] Oct 31 23:44:05.858015 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.857982 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-44381a-serving-cert\"" Oct 31 23:44:05.924533 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.924491 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:05.935619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.935587 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86667268-0fab-4537-b06e-16563d324da0" path="/var/lib/kubelet/pods/86667268-0fab-4537-b06e-16563d324da0/volumes" Oct 31 23:44:05.939314 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.939284 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqxm\" (UniqueName: \"kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:05.939468 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.939448 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:05.939565 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.939519 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:05.939664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.939641 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:05.954669 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.954635 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc"] Oct 31 23:44:05.958725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.958699 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:05.973252 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:05.973227 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc"] Oct 31 23:44:06.040594 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.040554 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.040764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.040600 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.040764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.040639 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl948\" (UniqueName: \"kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948\") pod \"payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.040764 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.040656 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.040957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.040832 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqxm\" (UniqueName: \"kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.040957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.040875 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.041342 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.041320 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.042955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.042932 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.043164 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.043143 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.052998 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.052977 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqxm\" (UniqueName: \"kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm\") pod \"maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.061222 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.061205 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-44381a\"" Oct 31 23:44:06.064749 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.064729 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:06.142333 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.142250 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl948\" (UniqueName: \"kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948\") pod \"payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.142486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.142360 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.143070 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.143050 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.151142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.151120 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:06.154990 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.154969 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl948\" (UniqueName: \"kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948\") pod \"payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.276433 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.276399 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:06.430283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.430258 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt"] Oct 31 23:44:06.432000 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:44:06.431971 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b8d8b9_bf4d_4b8f_b076_87612d3377d5.slice/crio-292adaff8c0bd08aa3e4a6df6f297e6fff0d06be4fec85c602d953cd04db4285 WatchSource:0}: Error finding container 292adaff8c0bd08aa3e4a6df6f297e6fff0d06be4fec85c602d953cd04db4285: Status 404 returned error can't find the container with id 292adaff8c0bd08aa3e4a6df6f297e6fff0d06be4fec85c602d953cd04db4285 Oct 31 23:44:06.503639 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.503602 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" event={"ID":"50b8d8b9-bf4d-4b8f-b076-87612d3377d5","Type":"ContainerStarted","Data":"292adaff8c0bd08aa3e4a6df6f297e6fff0d06be4fec85c602d953cd04db4285"} Oct 31 23:44:06.560723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.560692 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc"] Oct 31 23:44:06.562464 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:44:06.562433 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcacf878_770b_4dd8_afe7_e3f5b5179ddf.slice/crio-06a0d7befa25b77d2ed2fb63af686a438d04072560ed1f0b879acdafe0674a48 WatchSource:0}: Error finding container 06a0d7befa25b77d2ed2fb63af686a438d04072560ed1f0b879acdafe0674a48: Status 404 returned error can't find the container with id 06a0d7befa25b77d2ed2fb63af686a438d04072560ed1f0b879acdafe0674a48 Oct 31 23:44:06.709828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:06.709641 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:07.413803 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.413756 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:07.510916 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.510878 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" event={"ID":"50b8d8b9-bf4d-4b8f-b076-87612d3377d5","Type":"ContainerStarted","Data":"571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91"} Oct 31 23:44:07.511131 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.510995 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:07.512724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.512700 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" event={"ID":"bcacf878-770b-4dd8-afe7-e3f5b5179ddf","Type":"ContainerStarted","Data":"e6e8e49b4ce36ccd5bc1e5923cdd4e6814b93186108fd4dd79580a815c13a40a"} Oct 31 23:44:07.512856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.512742 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" event={"ID":"bcacf878-770b-4dd8-afe7-e3f5b5179ddf","Type":"ContainerStarted","Data":"06a0d7befa25b77d2ed2fb63af686a438d04072560ed1f0b879acdafe0674a48"} Oct 31 23:44:07.512856 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.512781 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:07.535241 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.535200 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" podStartSLOduration=2.105822376 podStartE2EDuration="2.535188318s" totalImagesPullingTime="429.365942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:44:07.530814233 +0000 UTC m=+3273.760368644" watchObservedRunningTime="2026-10-31 23:44:07.535188318 +0000 UTC m=+3273.764742728" Oct 31 23:44:07.554398 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:07.554349 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" podStartSLOduration=2.5543377229999997 podStartE2EDuration="2.554337723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 23:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:44:07.551494715 +0000 UTC m=+3273.781049125" watchObservedRunningTime="2026-10-31 23:44:07.554337723 +0000 UTC m=+3273.783892132" Oct 31 23:44:08.134941 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:08.132250 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:08.908318 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:08.908224 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:09.600466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:09.600420 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:10.309788 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:10.309753 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:10.456490 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:44:10.456451 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:44:10.971077 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:10.971041 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:11.547773 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:11.547742 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:12.104504 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:12.104467 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:12.708293 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:12.708258 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:13.304288 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:13.304249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:13.526801 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:13.526775 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:44:13.910307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:13.910271 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:14.563763 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:14.563710 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:15.214854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:15.214812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:15.805501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:15.805453 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:15.939717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:15.939679 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:16.151276 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:16.151202 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:16.516728 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:16.516688 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:17.111731 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:17.111700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:17.720908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:17.720871 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:18.304392 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:18.304353 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:18.525646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:18.525617 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:44:18.911819 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:18.911786 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:19.512466 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:19.512430 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:19.846187 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:19.846098 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:20.118969 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:20.118871 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:20.808948 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:20.808914 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:21.519089 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:21.519054 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:22.110849 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:22.110813 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:22.800817 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:22.800746 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:23.403438 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:23.403405 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:24.011405 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:24.011354 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:24.617038 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:24.617002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:25.227912 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:25.227881 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:25.909422 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:25.909386 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:26.510865 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:26.510825 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:27.122492 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:27.122457 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:27.717575 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:27.717508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:28.414710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:28.414656 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:29.093095 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:29.093036 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:29.709047 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:29.708941 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:30.296734 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:30.296699 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:30.993258 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:30.993228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:31.717019 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:31.716983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:32.336993 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:32.336946 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:32.906240 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:32.906204 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:33.614896 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:33.614866 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:34.220097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:34.220052 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:34.817679 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:34.817639 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:35.421563 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:35.421516 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:36.011953 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:36.011917 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:36.617488 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:36.617451 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:37.218506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:37.218402 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:37.805525 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:37.805493 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:38.308854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:38.308822 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:38.508200 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:38.508167 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:39.121532 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:39.121498 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:39.715309 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:39.715277 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:40.315018 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:40.314982 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:40.455177 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:44:40.455140 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:44:40.908161 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:40.908131 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:41.460157 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:41.460127 5043 scope.go:122] "RemoveContainer" containerID="e7e4c4f88dc4d865c4d2b3b7afa038637a7a19de0f63100c4e6d2afb9c8cb6ae" Oct 31 23:44:41.596008 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:41.595978 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:42.213075 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:42.213038 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:42.810389 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:42.810357 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:43.489962 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:43.489931 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:44.110717 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:44.110676 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:44.708156 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:44.708120 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:45.308893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:45.308856 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:45.911827 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:45.911773 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:46.602035 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:46.602002 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:47.212223 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:47.212186 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:47.799843 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:47.799811 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:48.409024 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:48.408976 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:49.002458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:49.002425 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:49.613970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:49.613926 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:50.204459 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:50.204422 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:50.807968 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:50.807933 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:51.403857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:51.403821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:52.026329 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:52.026290 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:52.608654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:52.608612 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:53.232416 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:53.232381 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:53.801659 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:53.801609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:54.430242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:54.430208 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:55.015296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:55.015259 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:55.600350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:55.600308 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:56.207708 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:56.207622 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:56.807538 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:56.807495 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:57.414651 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:57.414598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:58.012554 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:58.012520 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:58.612259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:58.612228 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:59.221857 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:59.221818 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:44:59.822903 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:44:59.822859 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:00.517119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:00.517074 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:01.288278 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:01.288198 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:01.897694 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:01.897649 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:02.624160 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:02.624121 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:03.024097 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:03.024064 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp"] Oct 31 23:45:03.028950 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:03.028920 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p7ccp"] Oct 31 23:45:03.219022 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:03.218983 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:03.934895 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:03.934859 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204c09bf-29f4-459e-8982-ced30c3e87b1" path="/var/lib/kubelet/pods/204c09bf-29f4-459e-8982-ced30c3e87b1/volumes" Oct 31 23:45:04.107391 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:04.107360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:04.718570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:04.718508 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:05.322653 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:05.322609 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:05.915578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:05.915516 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:06.526516 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:06.526409 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:07.286201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:07.286169 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:08.006307 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:08.006249 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:08.617370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:08.617301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:09.208451 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:09.208387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:09.808237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:09.808185 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:10.406113 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:10.406016 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:10.456055 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:45:10.456011 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:45:11.033216 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:11.033182 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:11.680236 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:11.680196 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:11.816630 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:11.816598 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:12.310109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:12.310073 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:12.913465 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:12.913387 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:13.513649 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:13.513615 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:14.209368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:14.209323 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:14.912300 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:14.912234 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:15.514224 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:15.514166 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:15.831478 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:15.831374 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:15.969839 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:15.969806 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:16.123495 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:16.123400 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:16.264705 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:16.264658 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:16.717899 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:16.717843 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:17.303299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:17.303244 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:17.910917 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:17.910883 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:18.509265 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:18.509224 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:19.111775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:19.111746 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:19.709942 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:19.709909 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:20.309124 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:20.309021 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:21.016037 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:21.015997 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:21.026742 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:21.026705 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn"] Oct 31 23:45:21.030880 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:21.030848 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-sh9cn"] Oct 31 23:45:21.616414 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:21.616382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:21.937368 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:21.937274 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ed33c1-9bf1-4018-bbd6-547f64dd84ad" path="/var/lib/kubelet/pods/49ed33c1-9bf1-4018-bbd6-547f64dd84ad/volumes" Oct 31 23:45:22.411142 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:22.411109 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:23.067620 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:23.067582 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:23.706647 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:23.706594 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:24.316879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:24.316841 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:24.903147 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:24.903110 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:25.517393 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:25.517360 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:26.105632 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:26.105589 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:26.707027 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:26.706995 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:27.309561 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:27.309501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:28.004246 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:28.004216 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:28.618723 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:28.618679 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:29.296291 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:29.296253 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:29.911375 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:29.911341 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:30.504934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:30.504887 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:31.172442 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:31.172406 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:31.706807 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:31.706766 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:32.300197 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:32.300164 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:32.905958 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:32.905922 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:33.498194 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:33.498161 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:34.123107 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:34.123072 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:34.705261 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:34.705215 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:35.299934 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:35.299899 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:35.906259 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:35.906225 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:36.510572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:36.510501 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:37.100419 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:37.100382 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:37.806854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:37.806783 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:38.410179 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:38.410143 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:39.008454 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:39.008412 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:39.621590 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:39.621537 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:40.393908 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:40.393868 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:40.457001 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:45:40.456965 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:45:41.012693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:41.012657 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:41.601845 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:41.601812 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:42.201007 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:42.200975 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:42.804486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:42.804445 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:43.398800 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:43.398757 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:44.000083 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:44.000048 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:44.626350 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:44.626301 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:45.210573 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:45.210519 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:45.960458 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:45.960422 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:46.616887 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:46.614090 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:47.215675 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:47.215633 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:47.810871 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:47.810830 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:48.407859 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:48.407821 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:49.021407 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:49.021371 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:49.613937 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:49.613900 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:49.965814 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:49.965727 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc"] Oct 31 23:45:50.103463 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:50.103427 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:50.235794 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:50.235709 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:54.966854 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:54.966806 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" podUID="bcacf878-770b-4dd8-afe7-e3f5b5179ddf" containerName="payload-pre-processing" containerID="cri-o://e6e8e49b4ce36ccd5bc1e5923cdd4e6814b93186108fd4dd79580a815c13a40a" gracePeriod=25 Oct 31 23:45:55.114643 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.113921 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt"] Oct 31 23:45:55.116971 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.116923 5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" podUID="50b8d8b9-bf4d-4b8f-b076-87612d3377d5" containerName="maas-api" containerID="cri-o://571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91" gracePeriod=30 Oct 31 23:45:55.161119 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.161083 5043 generic.go:356] "Generic (PLEG): container finished" podID="bcacf878-770b-4dd8-afe7-e3f5b5179ddf" containerID="e6e8e49b4ce36ccd5bc1e5923cdd4e6814b93186108fd4dd79580a815c13a40a" exitCode=0 Oct 31 23:45:55.161254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.161157 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" event={"ID":"bcacf878-770b-4dd8-afe7-e3f5b5179ddf","Type":"ContainerDied","Data":"e6e8e49b4ce36ccd5bc1e5923cdd4e6814b93186108fd4dd79580a815c13a40a"} Oct 31 23:45:55.195775 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.195714 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:55.277033 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.276829 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:45:55.342106 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.342006 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:55.413644 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.413604 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume\") pod \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " Oct 31 23:45:55.413809 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.413777 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948\" (UniqueName: \"kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948\") pod \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\" (UID: \"bcacf878-770b-4dd8-afe7-e3f5b5179ddf\") " Oct 31 23:45:55.413999 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.413978 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume" pod "bcacf878-770b-4dd8-afe7-e3f5b5179ddf" (UID: "bcacf878-770b-4dd8-afe7-e3f5b5179ddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:45:55.414165 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.414148 5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-plugins-config-volume\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:55.418574 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.418517 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948" pod "bcacf878-770b-4dd8-afe7-e3f5b5179ddf" (UID: "bcacf878-770b-4dd8-afe7-e3f5b5179ddf"). InnerVolumeSpecName "kube-api-access-nl948". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:45:55.419522 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.419499 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:45:55.489290 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.489201 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:55.515380 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.515348 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls\") pod \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " Oct 31 23:45:55.515572 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.515397 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs\") pod \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " Oct 31 23:45:55.515654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.515598 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm\" (UniqueName: \"kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm\") pod \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " Oct 31 23:45:55.515718 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.515652 5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca\") pod \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\" (UID: \"50b8d8b9-bf4d-4b8f-b076-87612d3377d5\") " Oct 31 23:45:55.516073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.516051 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl948\" (UniqueName: \"kubernetes.io/projected/bcacf878-770b-4dd8-afe7-e3f5b5179ddf-kube-api-access-nl948\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:55.516437 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.516411 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca" pod "50b8d8b9-bf4d-4b8f-b076-87612d3377d5" (UID: "50b8d8b9-bf4d-4b8f-b076-87612d3377d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 23:45:55.517684 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.517656 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls" pod "50b8d8b9-bf4d-4b8f-b076-87612d3377d5" (UID: "50b8d8b9-bf4d-4b8f-b076-87612d3377d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:45:55.517876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.517851 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs" pod "50b8d8b9-bf4d-4b8f-b076-87612d3377d5" (UID: "50b8d8b9-bf4d-4b8f-b076-87612d3377d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 23:45:55.518312 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.518290 5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm" pod "50b8d8b9-bf4d-4b8f-b076-87612d3377d5" (UID: "50b8d8b9-bf4d-4b8f-b076-87612d3377d5"). InnerVolumeSpecName "kube-api-access-dcqxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 23:45:55.617046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.617016 5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-maas-api-tls\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:55.617046 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.617047 5043 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-metrics-certs\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:55.617283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.617061 5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqxm\" (UniqueName: \"kubernetes.io/projected/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-kube-api-access-dcqxm\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:55.617283 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.617076 5043 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b8d8b9-bf4d-4b8f-b076-87612d3377d5-openshift-service-ca\") on node \"ip-10-0-133-222.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 23:45:55.922994 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:55.922943 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:45:56.169150 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.169118 5043 generic.go:356] "Generic (PLEG): container finished" podID="50b8d8b9-bf4d-4b8f-b076-87612d3377d5" containerID="571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91" exitCode=0 Oct 31 23:45:56.169655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.169176 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" Oct 31 23:45:56.169655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.169202 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" event={"ID":"50b8d8b9-bf4d-4b8f-b076-87612d3377d5","Type":"ContainerDied","Data":"571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91"} Oct 31 23:45:56.169655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.169263 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt" event={"ID":"50b8d8b9-bf4d-4b8f-b076-87612d3377d5","Type":"ContainerDied","Data":"292adaff8c0bd08aa3e4a6df6f297e6fff0d06be4fec85c602d953cd04db4285"} Oct 31 23:45:56.169655 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.169294 5043 scope.go:122] "RemoveContainer" containerID="571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91" Oct 31 23:45:56.170957 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.170936 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" event={"ID":"bcacf878-770b-4dd8-afe7-e3f5b5179ddf","Type":"ContainerDied","Data":"06a0d7befa25b77d2ed2fb63af686a438d04072560ed1f0b879acdafe0674a48"} Oct 31 23:45:56.171069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.170969 5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc" Oct 31 23:45:56.179828 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.179809 5043 scope.go:122] "RemoveContainer" containerID="571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91" Oct 31 23:45:56.180108 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:45:56.180085 5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91\": container with ID starting with 571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91 not found: ID does not exist" containerID="571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91" Oct 31 23:45:56.180191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.180120 5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91"} err="failed to get container status \"571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91\": rpc error: code = NotFound desc = could not find container \"571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91\": container with ID starting with 571352d229bb596ee3abfd383180222e8562a5cdb8a1c4ee4eb974d746be9d91 not found: ID does not exist" Oct 31 23:45:56.180191 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.180143 5043 scope.go:122] "RemoveContainer" containerID="e6e8e49b4ce36ccd5bc1e5923cdd4e6814b93186108fd4dd79580a815c13a40a" Oct 31 23:45:56.189506 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.189486 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt"] Oct 31 23:45:56.193338 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.193318 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-44381a-7865d845fc-hr5tt"] Oct 31 23:45:56.208295 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.208274 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc"] Oct 31 23:45:56.213606 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:56.213584 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-44381a-9c75f4dc-8kvxc"] Oct 31 23:45:57.935282 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:57.935241 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b8d8b9-bf4d-4b8f-b076-87612d3377d5" path="/var/lib/kubelet/pods/50b8d8b9-bf4d-4b8f-b076-87612d3377d5/volumes" Oct 31 23:45:57.935832 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:45:57.935812 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcacf878-770b-4dd8-afe7-e3f5b5179ddf" path="/var/lib/kubelet/pods/bcacf878-770b-4dd8-afe7-e3f5b5179ddf/volumes" Oct 31 23:46:02.040970 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:02.040937 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:02.180913 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:02.180873 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:02.401473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:02.401391 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:07.863842 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:07.863796 5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-smff4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 23:46:07.864237 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:07.863870 5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-smff4" podUID="7578ea20-3979-4843-98cf-78864e3774b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 23:46:10.470097 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:46:10.470056 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:46:15.996724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:15.996686 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:16.259439 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:16.259350 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:17.786447 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:17.786416 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:46:17.791757 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:17.791734 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:46:17.801519 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:17.801501 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:46:19.108598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:19.108572 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:46:19.113906 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:19.113881 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:46:19.124173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:19.124152 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-2tbhj_c8c42546-d756-4965-96c3-e3e05dd3d89b/csi-driver/0.log" Oct 31 23:46:25.023619 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:25.023509 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj"] Oct 31 23:46:25.026248 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:25.026218 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92910a18-xhwmj"] Oct 31 23:46:25.938946 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:25.938903 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69beb65f-fee0-4fcd-a000-a36b6f45ac4b" path="/var/lib/kubelet/pods/69beb65f-fee0-4fcd-a000-a36b6f45ac4b/volumes" Oct 31 23:46:40.455947 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:46:40.455906 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:46:41.495732 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:41.495694 5043 scope.go:122] "RemoveContainer" containerID="5701a25140ef4e3fed2122665e641fc73ed3f92a337ddde66dcb7aaf6b9a69b9" Oct 31 23:46:41.505069 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:41.505048 5043 scope.go:122] "RemoveContainer" containerID="9eac086b493425f591d040d7ac51a734a6efa3d13a3be7ae59a3c9430844eb9d" Oct 31 23:46:47.542735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:47.542700 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:49.075233 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:49.075199 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:46:52.864986 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:46:52.864951 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bfb984676-kv9dg_d55a9c0b-0e57-4c80-b2e2-929b74863635/authorino/0.log" Oct 31 23:47:03.763521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:03.763486 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4vgvf_677d053b-fbf4-4b76-9782-6b3235dc340f/manager/1.log" Oct 31 23:47:05.271753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:05.271722 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4vgvf_677d053b-fbf4-4b76-9782-6b3235dc340f/manager/0.log" Oct 31 23:47:06.027693 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:06.027667 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bf6d56f9f-hqjpf_8e2ae269-d7ca-4d38-8d6a-d348a4f450be/maas-api/0.log" Oct 31 23:47:06.388652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:06.388575 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-338bf790-mqjd7_c6a08599-6bca-4f6a-be22-d7ea314ed744/revoke-keys/0.log" Oct 31 23:47:06.563145 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:06.563115 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-768xw_4e964ec2-2f8c-4295-9698-9b129f2e41c5/revoke-keys/0.log" Oct 31 23:47:06.730473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:06.730402 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-76wcf_ada9a6e4-9b55-4945-9a83-9e5984df04d7/revoke-keys/0.log" Oct 31 23:47:08.797810 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:08.797779 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bfb984676-kv9dg_d55a9c0b-0e57-4c80-b2e2-929b74863635/authorino/0.log" Oct 31 23:47:09.233714 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:09.233628 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ss2sd_b4fd03ee-af94-4df1-b834-eced676d664e/manager/1.log" Oct 31 23:47:09.615359 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:09.615308 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-xdnsr_cd2d832b-995b-41b4-b9c5-8372b2ec54a5/manager/0.log" Oct 31 23:47:10.035940 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:10.035911 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ktbm2_485a8e4b-a74c-4463-affc-44a6374fb81c/manager/1.log" Oct 31 23:47:10.442988 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:10.442910 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ss2sd_b4fd03ee-af94-4df1-b834-eced676d664e/manager/0.log" Oct 31 23:47:10.456726 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:47:10.456691 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:47:10.645177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:10.645148 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ktbm2_485a8e4b-a74c-4463-affc-44a6374fb81c/manager/0.log" Oct 31 23:47:11.201242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:11.201214 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7_0f223a32-eac0-491e-ae99-0330a8043011/istio-proxy/0.log" Oct 31 23:47:11.963029 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:11.962996 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/1.log" Oct 31 23:47:12.337779 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:12.337754 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-fw9np_c08d1d39-5bdc-4834-a0de-af51ef202a20/istio-proxy/0.log" Oct 31 23:47:12.705902 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:12.705822 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/1.log" Oct 31 23:47:13.378710 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:13.378675 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/1.log" Oct 31 23:47:14.589530 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:14.589461 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/1.log" Oct 31 23:47:15.034955 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:15.034927 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/0.log" Oct 31 23:47:15.600821 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:15.600790 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/0.log" Oct 31 23:47:15.772738 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:15.772708 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/0.log" Oct 31 23:47:16.051646 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:16.051617 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:47:16.205517 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:16.205483 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:47:16.331243 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:16.331177 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:47:17.056825 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:17.056798 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ld9pm_98438769-91a9-4aec-b9e9-d0c4044cae37/main/0.log" Oct 31 23:47:17.409321 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:17.409246 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2859g_f091b8de-14bb-4d9d-ac75-3edd31cb3e68/main/0.log" Oct 31 23:47:17.779625 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:17.779596 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-55lxc_9cf1ba00-1cc7-4af1-a5df-efeafadaac9b/main/0.log" Oct 31 23:47:17.962830 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:17.962803 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-spffz_d63e791b-7230-4884-bdba-3996f4205f60/main/0.log" Oct 31 23:47:19.058195 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:19.058168 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bfb984676-kv9dg_d55a9c0b-0e57-4c80-b2e2-929b74863635/authorino/0.log" Oct 31 23:47:28.675006 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:28.674973 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4vgvf_677d053b-fbf4-4b76-9782-6b3235dc340f/manager/1.log" Oct 31 23:47:30.435581 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:30.435532 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bf6d56f9f-hqjpf_8e2ae269-d7ca-4d38-8d6a-d348a4f450be/maas-api/0.log" Oct 31 23:47:30.827484 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:30.827454 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-338bf790-mqjd7_c6a08599-6bca-4f6a-be22-d7ea314ed744/revoke-keys/0.log" Oct 31 23:47:31.014528 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:31.014498 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-768xw_4e964ec2-2f8c-4295-9698-9b129f2e41c5/revoke-keys/0.log" Oct 31 23:47:31.024201 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:31.024176 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf"] Oct 31 23:47:31.027306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:31.027272 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-76wcf"] Oct 31 23:47:31.935787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:31.935755 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada9a6e4-9b55-4945-9a83-9e5984df04d7" path="/var/lib/kubelet/pods/ada9a6e4-9b55-4945-9a83-9e5984df04d7/volumes" Oct 31 23:47:32.810430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:32.810401 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bfb984676-kv9dg_d55a9c0b-0e57-4c80-b2e2-929b74863635/authorino/0.log" Oct 31 23:47:33.223890 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:33.223802 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ss2sd_b4fd03ee-af94-4df1-b834-eced676d664e/manager/1.log" Oct 31 23:47:33.590109 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:33.590083 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-xdnsr_cd2d832b-995b-41b4-b9c5-8372b2ec54a5/manager/0.log" Oct 31 23:47:33.958570 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:33.958475 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ktbm2_485a8e4b-a74c-4463-affc-44a6374fb81c/manager/1.log" Oct 31 23:47:34.521242 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:34.521213 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7_0f223a32-eac0-491e-ae99-0330a8043011/istio-proxy/0.log" Oct 31 23:47:35.266943 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:35.266908 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/1.log" Oct 31 23:47:35.637356 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:35.637275 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-fw9np_c08d1d39-5bdc-4834-a0de-af51ef202a20/istio-proxy/0.log" Oct 31 23:47:35.991308 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:35.991286 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/1.log" Oct 31 23:47:36.513254 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:36.513225 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/1.log" Oct 31 23:47:37.584725 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:37.584695 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/1.log" Oct 31 23:47:38.290660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:38.290624 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-ld9pm_98438769-91a9-4aec-b9e9-d0c4044cae37/main/0.log" Oct 31 23:47:38.657984 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:38.657895 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-2859g_f091b8de-14bb-4d9d-ac75-3edd31cb3e68/main/0.log" Oct 31 23:47:39.027296 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:39.027266 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-55lxc_9cf1ba00-1cc7-4af1-a5df-efeafadaac9b/main/0.log" Oct 31 23:47:39.218238 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:39.218208 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-spffz_d63e791b-7230-4884-bdba-3996f4205f60/main/0.log" Oct 31 23:47:40.458143 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:47:40.458099 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:47:41.540411 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:41.540367 5043 scope.go:122] "RemoveContainer" containerID="d619f65ff15370c4e0683f6f26ba18f8aac10be471f41e7b7fe92718235cc1ec" Oct 31 23:47:41.549473 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:41.549446 5043 scope.go:122] "RemoveContainer" containerID="017b50046e2f293e1dd0916df4908d9292a3a136d9ff4519f2d4660460d7c415" Oct 31 23:47:47.024879 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:47.024837 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw"] Oct 31 23:47:47.029196 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:47.029163 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-768xw"] Oct 31 23:47:47.934051 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:47:47.934014 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e964ec2-2f8c-4295-9698-9b129f2e41c5" path="/var/lib/kubelet/pods/4e964ec2-2f8c-4295-9698-9b129f2e41c5/volumes" Oct 31 23:48:08.026521 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:08.026481 5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7"] Oct 31 23:48:08.029144 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:08.029119 5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338bf790-mqjd7"] Oct 31 23:48:09.936429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:09.936391 5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a08599-6bca-4f6a-be22-d7ea314ed744" path="/var/lib/kubelet/pods/c6a08599-6bca-4f6a-be22-d7ea314ed744/volumes" Oct 31 23:48:10.456742 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:48:10.456707 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:48:16.101924 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:16.101889 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:48:16.246501 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:16.246466 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:48:40.455614 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:48:40.455568 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:48:41.573862 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:41.573833 5043 scope.go:122] "RemoveContainer" containerID="7f892a7a9b36b36e8da769cdb7e74b193d9602c41d99e22b86cb2b99c8fe7d8f" Oct 31 23:48:51.259173 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:51.259137 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:48:54.411334 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:54.411297 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:48:54.670363 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:54.670277 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:48:57.746654 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:57.746615 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xrfs4_ce8955fd-a862-461a-bce9-a5be688358ef/global-pull-secret-syncer/0.log" Oct 31 23:48:57.802724 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:57.802675 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gftjj_3ce54792-3b37-4895-a39a-d233a7e3d4ee/connection-checker/0.log" Oct 31 23:48:57.835475 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:57.835447 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k5t2f_e4043168-4d17-4609-bc2f-37951b7480b4/connection-checker/0.log" Oct 31 23:48:57.864486 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:57.864464 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-khbjk_6c14a7d4-f9a1-4e58-8ee5-8455aecf175e/connection-checker/0.log" Oct 31 23:48:57.910947 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:57.910917 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" Oct 31 23:48:58.076317 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:48:58.076239 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-222.us-east-2.compute.internal_a70adfcc128189ec0c0f585313e4a644/haproxy/0.log" Oct 31 23:49:04.875320 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:04.875285 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bfb984676-kv9dg_d55a9c0b-0e57-4c80-b2e2-929b74863635/authorino/0.log" Oct 31 23:49:04.962063 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:04.962035 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ss2sd_b4fd03ee-af94-4df1-b834-eced676d664e/manager/1.log" Oct 31 23:49:04.962275 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:04.962201 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ss2sd_b4fd03ee-af94-4df1-b834-eced676d664e/manager/0.log" Oct 31 23:49:05.040735 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:05.040698 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-xdnsr_cd2d832b-995b-41b4-b9c5-8372b2ec54a5/manager/0.log" Oct 31 23:49:05.120674 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:05.120633 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ktbm2_485a8e4b-a74c-4463-affc-44a6374fb81c/manager/0.log" Oct 31 23:49:05.132799 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:05.132735 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ktbm2_485a8e4b-a74c-4463-affc-44a6374fb81c/manager/1.log" Oct 31 23:49:06.668893 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:06.668864 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2l28w_8dfd0ee8-e42a-4bbd-b355-44262861da53/cluster-monitoring-operator/0.log" Oct 31 23:49:06.959904 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:06.959824 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kxz5_9c0cbba8-7d58-4923-80ea-55ab6ce5fe70/node-exporter/0.log" Oct 31 23:49:06.981787 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:06.981759 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kxz5_9c0cbba8-7d58-4923-80ea-55ab6ce5fe70/kube-rbac-proxy/0.log" Oct 31 23:49:07.016864 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.016840 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kxz5_9c0cbba8-7d58-4923-80ea-55ab6ce5fe70/init-textfile/0.log" Oct 31 23:49:07.376623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.376592 5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l"] Oct 31 23:49:07.377071 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.377059 5043 cpu_manager.go:414] "containerMap: removing container" podUID="bcacf878-770b-4dd8-afe7-e3f5b5179ddf" containerName="payload-pre-processing" Oct 31 23:49:07.377117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.377073 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcacf878-770b-4dd8-afe7-e3f5b5179ddf" containerName="payload-pre-processing" Oct 31 23:49:07.377117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.377088 5043 cpu_manager.go:414] "containerMap: removing container" podUID="50b8d8b9-bf4d-4b8f-b076-87612d3377d5" containerName="maas-api" Oct 31 23:49:07.377117 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.377096 5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b8d8b9-bf4d-4b8f-b076-87612d3377d5" containerName="maas-api" Oct 31 23:49:07.377219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.377169 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcacf878-770b-4dd8-afe7-e3f5b5179ddf" containerName="payload-pre-processing" Oct 31 23:49:07.377219 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.377180 5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b8d8b9-bf4d-4b8f-b076-87612d3377d5" containerName="maas-api" Oct 31 23:49:07.380370 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.380347 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.391487 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.391458 5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nmqzm\"/\"default-dockercfg-sstqk\"" Oct 31 23:49:07.391660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.391484 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmqzm\"/\"openshift-service-ca.crt\"" Oct 31 23:49:07.391660 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.391580 5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nmqzm\"/\"kube-root-ca.crt\"" Oct 31 23:49:07.395169 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.395142 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l"] Oct 31 23:49:07.470158 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.470130 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-lib-modules\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.470310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.470166 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-sys\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.470310 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.470274 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-proc\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.470429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.470309 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-podres\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.470429 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.470325 5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sn5\" (UniqueName: \"kubernetes.io/projected/98128de3-09c2-4957-8970-4e3fb21c4794-kube-api-access-j9sn5\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571225 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571195 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-proc\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571231 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-podres\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571250 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sn5\" (UniqueName: \"kubernetes.io/projected/98128de3-09c2-4957-8970-4e3fb21c4794-kube-api-access-j9sn5\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571316 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-proc\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571341 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-lib-modules\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571388 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571369 5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-sys\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571398 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-lib-modules\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571338 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-podres\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.571627 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.571470 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98128de3-09c2-4957-8970-4e3fb21c4794-sys\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.583410 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.583380 5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sn5\" (UniqueName: \"kubernetes.io/projected/98128de3-09c2-4957-8970-4e3fb21c4794-kube-api-access-j9sn5\") pod \"perf-node-gather-daemonset-kb96l\" (UID: \"98128de3-09c2-4957-8970-4e3fb21c4794\") " pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.704445 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.704378 5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:07.983176 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:07.983150 5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l"] Oct 31 23:49:07.984969 ip-10-0-133-222 kubenswrapper[5043]: W1031 23:49:07.984943 5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98128de3_09c2_4957_8970_4e3fb21c4794.slice/crio-7ca677ed24932f0288ed83e612fd8a35eb0f887783dec458b95d61720bcef873 WatchSource:0}: Error finding container 7ca677ed24932f0288ed83e612fd8a35eb0f887783dec458b95d61720bcef873: Status 404 returned error can't find the container with id 7ca677ed24932f0288ed83e612fd8a35eb0f887783dec458b95d61720bcef873 Oct 31 23:49:08.457472 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:08.457423 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" event={"ID":"98128de3-09c2-4957-8970-4e3fb21c4794","Type":"ContainerStarted","Data":"606761b111d997a6011e92982afec3d4876ee9aac9641b2b0608abf27737d893"} Oct 31 23:49:08.457673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:08.457486 5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" event={"ID":"98128de3-09c2-4957-8970-4e3fb21c4794","Type":"ContainerStarted","Data":"7ca677ed24932f0288ed83e612fd8a35eb0f887783dec458b95d61720bcef873"} Oct 31 23:49:08.457673 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:08.457608 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:08.476623 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:08.476580 5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" podStartSLOduration=1.4765673000000001 podStartE2EDuration="1.4765673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 23:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 23:49:08.475909065 +0000 UTC m=+3574.705463476" watchObservedRunningTime="2026-10-31 23:49:08.4765673 +0000 UTC m=+3574.706121710" Oct 31 23:49:09.055049 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:09.055018 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-qv9l7_eec52f11-3785-48d2-92a1-cbd30a6047e2/networking-console-plugin/0.log" Oct 31 23:49:09.655753 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:09.655718 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/3.log" Oct 31 23:49:09.659670 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:09.659648 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-vkldg_12ea1ea8-83a9-4c99-a124-5bf66d1f26b5/console-operator/2.log" Oct 31 23:49:10.455305 ip-10-0-133-222 kubenswrapper[5043]: E1031 23:49:10.455272 5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtb2p" containerName="extractor" Oct 31 23:49:10.817299 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:10.817268 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-q7688_2d1c440e-90c0-4411-84a4-a5bfaf91d3e9/volume-data-source-validator/0.log" Oct 31 23:49:12.133177 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:12.133150 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xctjg_67c4f288-6571-456f-8aa8-d6f5947eb14d/dns/0.log" Oct 31 23:49:12.159306 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:12.159288 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xctjg_67c4f288-6571-456f-8aa8-d6f5947eb14d/kube-rbac-proxy/0.log" Oct 31 23:49:12.222882 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:12.222862 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sk99x_13e8e748-f088-43d1-8011-bf6c8f91ec59/dns-node-resolver/0.log" Oct 31 23:49:12.923687 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:12.923658 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-58lh2_37b0f64a-6235-4472-b7e2-a2d2a08fd388/node-ca/0.log" Oct 31 23:49:13.970287 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:13.970254 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7_0f223a32-eac0-491e-ae99-0330a8043011/istio-proxy/0.log" Oct 31 23:49:14.475640 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:14.475611 5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nmqzm/perf-node-gather-daemonset-kb96l" Oct 31 23:49:15.895526 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:15.895496 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/1.log" Oct 31 23:49:15.895931 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:15.895681 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/0.log" Oct 31 23:49:15.957199 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:15.957175 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-fw9np_c08d1d39-5bdc-4834-a0de-af51ef202a20/istio-proxy/0.log" Oct 31 23:49:16.025664 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.025638 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/1.log" Oct 31 23:49:16.025868 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.025846 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/0.log" Oct 31 23:49:16.132379 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.132347 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/1.log" Oct 31 23:49:16.132641 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.132623 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/0.log" Oct 31 23:49:16.175580 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.175476 5043 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpmrk_5a940d2c-fc2c-40ae-a8bd-5916089eb925/konnectivity-agent/0.log" limit=1024 Oct 31 23:49:16.362598 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.362566 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/1.log" Oct 31 23:49:16.363057 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:16.363039 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:49:17.885056 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:17.885024 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7_0f223a32-eac0-491e-ae99-0330a8043011/istio-proxy/0.log" Oct 31 23:49:18.563702 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:18.563673 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2hnp_f9191bfb-6d65-4d74-8948-fcdb9ad57b76/serve-healthcheck-canary/0.log" Oct 31 23:49:19.258578 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.258534 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fj2lt_aad03e4e-7139-463f-baf7-752544c6763f/insights-operator/0.log" Oct 31 23:49:19.260876 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.260850 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fj2lt_aad03e4e-7139-463f-baf7-752544c6763f/insights-operator/1.log" Oct 31 23:49:19.468657 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.468620 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtb2p_756d759c-3587-4ff9-9abb-f24c46f64077/kube-rbac-proxy/0.log" Oct 31 23:49:19.501002 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.500971 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtb2p_756d759c-3587-4ff9-9abb-f24c46f64077/exporter/0.log" Oct 31 23:49:19.529493 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.529421 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtb2p_756d759c-3587-4ff9-9abb-f24c46f64077/extractor/0.log" Oct 31 23:49:19.769713 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.769687 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/1.log" Oct 31 23:49:19.769881 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.769840 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-62pjc_39400b70-1b3e-4c2c-9ab8-a45e349837ad/istio-proxy/0.log" Oct 31 23:49:19.802652 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.802567 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-fw9np_c08d1d39-5bdc-4834-a0de-af51ef202a20/istio-proxy/0.log" Oct 31 23:49:19.838204 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.838164 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/1.log" Oct 31 23:49:19.838378 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.838215 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-l992j_d52de0ae-1237-4099-90e1-a65f810ea3b4/istio-proxy/0.log" Oct 31 23:49:19.899816 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.899787 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/1.log" Oct 31 23:49:19.900020 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:19.899977 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-smff4_7578ea20-3979-4843-98cf-78864e3774b6/istio-proxy/0.log" Oct 31 23:49:20.005349 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:20.005318 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/1.log" Oct 31 23:49:20.005789 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:20.005775 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8f9bc98f7-xn5rp_7afda5df-8528-4304-aa1d-156aefde0156/router/0.log" Oct 31 23:49:22.066023 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:22.065993 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-ftgld_0063f042-8457-40ab-97d7-94d1420af56b/manager/1.log" Oct 31 23:49:22.066524 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:22.066505 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-ftgld_0063f042-8457-40ab-97d7-94d1420af56b/manager/0.log" Oct 31 23:49:22.087430 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:22.087412 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c5c6d88b7-ftgld_0063f042-8457-40ab-97d7-94d1420af56b/copy-manifests/0.log" Oct 31 23:49:22.796073 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:22.796041 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5bkbx7_0f223a32-eac0-491e-ae99-0330a8043011/istio-proxy/0.log" Oct 31 23:49:23.373397 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:23.373366 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-64f549b4b7-q5w9p_863f9cb3-dbae-4b56-b240-eb661de1f2dc/manager/1.log" Oct 31 23:49:23.373853 ip-10-0-133-222 kubenswrapper[5043]: I1031 23:49:23.373651 5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-64f549b4b7-q5w9p_863f9cb3-dbae-4b56-b240-eb661de1f2dc/manager/0.log"